Oct 30 23:03:42.825327 ip-10-0-130-205 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 23:03:42.825336 ip-10-0-130-205 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 23:03:42.825343 ip-10-0-130-205 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 23:03:42.825562 ip-10-0-130-205 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 23:03:53.074877 ip-10-0-130-205 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 23:03:53.074891 ip-10-0-130-205 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c2f5b0afff5e41078a87a1566c9b40f4 --
Oct 30 23:05:41.429891 ip-10-0-130-205 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 23:05:41.842166 ip-10-0-130-205 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:05:41.842166 ip-10-0-130-205 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 23:05:41.842166 ip-10-0-130-205 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:05:41.842166 ip-10-0-130-205 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:05:41.846478 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846398    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:05:41.846478 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846479    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846484    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846487    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846489    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846492    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846495    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846497    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846500    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846503    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846505    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846513    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846516    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846518    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846521    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846523    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846526    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846529    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846531    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846534    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:05:41.846547 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846536    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846539    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846541    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846544    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846547    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846549    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846552    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846555    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846557    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846560    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846563    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846565    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846568    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846570    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846573    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846575    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846578    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846580    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846583    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846586    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:05:41.847044 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846588    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846591    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846607    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846611    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846613    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846615    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846618    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846621    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846623    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846626    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846628    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846631    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846634    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846637    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846640    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846643    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846645    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846648    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846651    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846653    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:05:41.847511 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846656    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846658    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846661    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846664    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846666    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846669    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846671    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846674    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846676    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846679    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846681    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846683    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846686    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846689    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846692    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846694    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846697    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846700    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846702    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:05:41.848030 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846705    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846707    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846710    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846713    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846716    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846718    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846721    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846724    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846726    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846728    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846731    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846733    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846736    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846739    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846741    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846744    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846746    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846749    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846751    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:05:41.848578 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846755    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846759    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846763    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846766    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846769    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846771    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846776    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.846780    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847575    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847583    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847586    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847589    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847607    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847610    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847614    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847617    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847619    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847623    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847626    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847629    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:05:41.849061 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847632    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847634    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847637    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847640    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847642    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847645    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847648    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847650    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847653    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847656    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847658    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847661    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847663    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847666    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847668    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847671    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847674    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847678    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847681    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:05:41.849534 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847683    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847687    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847690    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847693    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847696    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847698    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847701    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847703    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847706    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847709    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847711    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847714    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847717    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847719    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847722    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847725    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847727    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847732    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847736    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:05:41.850008 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847739    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847742    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847745    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847747    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847750    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847752    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847755    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847757    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847759    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847762    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847765    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847767    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847770    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847772    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847774    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847777    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847780    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847783    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847785    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847788    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:05:41.850473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847790    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847793    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847796    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847799    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847801    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847805    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847807    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847810    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847813    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847815    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847817    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847820    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847823    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847825    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847827    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847830    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847832    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847835    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847837    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847840    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:05:41.850998 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847842    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847844    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847847    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847850    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847852    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847855    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847858    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847860    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847863    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847866    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847869    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847871    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847874    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847876    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847880    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.847882    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848420    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848427    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848431    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848435    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:05:41.851470 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848438    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848441    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848443    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848446    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848449    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848452    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848454    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848457    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848459    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848462    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848465    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848467    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848470    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848473    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848477    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848481    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848484    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848487    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848490    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:05:41.851962 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848492    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848495    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848498    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848501    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848503    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848506    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848508    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848511    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848514    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848516    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848519    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848522    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848524    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848526    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848529    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848531    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848534    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848537    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848539    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:05:41.852414 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848542    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848544    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848547    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848549    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848552    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848554    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848566    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848569    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848571    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848578    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848580    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848583    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848586    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848588    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848608    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848612    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848617    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848621    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848623    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848626    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:05:41.852886 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848629    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848631    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848634    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848637    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848640    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848642    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848645    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848647    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848650    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848652    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848655    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848657    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848660    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848662    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848665    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848667    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848670    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848672    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848675    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:05:41.853379 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848677    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848680    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848684    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848686    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848689    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848691    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848694    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848696    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848699    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848702    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848704    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848707    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848709    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848712    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848714    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848717    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848720    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848722    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848725    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848727    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:05:41.853891 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848729    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848732    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848734    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848737    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.848740    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849756    2573 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849765    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849772    2573 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849776    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849781    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849784    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849788    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849793    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849796    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849799    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849803    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849806    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849817    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849820    2573 flags.go:64] FLAG: --cgroup-root=""
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849823    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849826    2573 flags.go:64] FLAG: --client-ca-file=""
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849829    2573 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849833    2573 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 23:05:41.854356 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849837    2573 flags.go:64] FLAG: --cluster-domain=""
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849840    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849843    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849846    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849849    2573 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849854    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849857    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849860    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849863    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849866    2573 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849869    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849872    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849875    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849877    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849881    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849884    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849887    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849890    2573 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849893    2573 flags.go:64] FLAG: --enable-server="true"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849896    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849901    2573 flags.go:64] FLAG: --event-burst="100"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849904    2573 flags.go:64] FLAG: --event-qps="50"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849907    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849910    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 23:05:41.854911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849913    2573 flags.go:64] FLAG: --eviction-hard=""
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849917    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849920    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849923    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849926    2573 flags.go:64] FLAG: --eviction-soft=""
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849929    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849934    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849937    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849941    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849944    2573 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849947    2573 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849949    2573 flags.go:64] FLAG: --feature-gates=""
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849953    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849956    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849959    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849962    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849965    2573 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849968    2573 flags.go:64] FLAG: --help="false"
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849971    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849975    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849977    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849980    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849984    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 23:05:41.855537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849987    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849990    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849992    2573 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849995    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.849998    2573 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850001    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850004    2573 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850007    2573 flags.go:64] FLAG: --kube-reserved=""
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850010    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850013    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850016    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850019    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850022    2573 flags.go:64] FLAG: --lock-file=""
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850025    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850027    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850031    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850037    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850040    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850043    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850046    2573 flags.go:64] FLAG: --logging-format="text"
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850049    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850053    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850055    2573 flags.go:64] FLAG: --manifest-url=""
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850058    2573 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850062    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 23:05:41.856152 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850065    2573 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850069    2573 flags.go:64] FLAG: --max-pods="110"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850072    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850075    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850078    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850081    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850084    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850087    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850090    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850096    2573 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850100    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850103    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850106    2573 flags.go:64] FLAG: --pod-cidr=""
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850108    2573 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850111    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850114    2573 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850117    2573 flags.go:64] FLAG: --port="10250"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850120    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850123    2573 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-02b7cc9bbc40822f3"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850125    2573 flags.go:64] FLAG: --qos-reserved=""
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850128    2573 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850131    2573 flags.go:64] FLAG: --register-node="true"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850134    2573 flags.go:64] FLAG: --register-with-taints=""
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850141    2573 flags.go:64] FLAG: --registry-burst="10"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850144    2573 flags.go:64] FLAG: --registry-qps="5"
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850146    2573 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 23:05:41.856775 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850150    2573 flags.go:64] FLAG: --reserved-memory=""
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850154    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850157    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850160    2573 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850162    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850165    2573 flags.go:64] FLAG: --runonce="false"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850168    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850171    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850174    2573 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850176    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850180    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850183    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850186    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850189    2573 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850191    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850194    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850197    2573 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850201    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850204    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850207    2573 flags.go:64] FLAG: --system-cgroups=""
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850210    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850215    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850218    2573 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850221    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850225    2573 flags.go:64] FLAG: --tls-min-version=""
Oct 30 23:05:41.857391 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850228    2573 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850230    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850233    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850236    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850239    2573 flags.go:64] FLAG: --v="2"
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850243    2573 flags.go:64] FLAG: --version="false"
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850247    2573 flags.go:64] FLAG: --vmodule=""
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850251    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850255    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850417    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850422    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850425    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850428    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850431    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850433    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850436    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850439    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850441    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850444    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850446    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850449    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850451    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:05:41.858240 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850454    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850457    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850460    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850463    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850465    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850468    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850472    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850476    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850479    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850481    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850484    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850487    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850489    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850492    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850495    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850498    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850500    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850503    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850505    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:05:41.858919 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850508    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850510    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850513    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850515    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850518    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850520    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850523    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850525    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850528    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850530    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850532    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850535    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850537    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850540    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850542    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850546    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850548    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850551    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850553    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850556    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:05:41.859372 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850558    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850561    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850563    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850566    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850568    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850573    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850575    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850578    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850580    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850582    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850585    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850588    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850590    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850606    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850609    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850612    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850614    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850617    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850619    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:05:41.859873 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850621    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850624    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850626    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850629    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850631    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850634    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850636    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850639    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850643    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850646    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850648    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850651    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850654    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850656    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850659    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850661    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850664    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850667    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850670    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:05:41.860364 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850673    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:05:41.860926 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850675    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:05:41.860926 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850678    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:05:41.860926 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850680    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:05:41.860926 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850683    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:05:41.860926 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850685    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:05:41.860926 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850688    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:05:41.860926 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850690    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:05:41.860926 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850693    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:05:41.860926 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850696    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:05:41.860926 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850698    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:05:41.860926 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850701    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:05:41.860926 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850703    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:05:41.860926 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850707    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:05:41.860926 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850711    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:05:41.860926 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850713    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:05:41.861281 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.850740    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:05:41.861281 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850782    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:05:41.861281 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850786    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:05:41.861281 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850789    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:05:41.861281 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850791    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:05:41.861281 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850794    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:05:41.861281 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850798    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:05:41.861281 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850800    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:05:41.861281 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850803    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:05:41.861281 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850805    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:05:41.861281 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850808    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:05:41.861281 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850810    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:05:41.861281 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850812    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:05:41.861281 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850815    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:05:41.861281 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850817    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:05:41.861281 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850821    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850825    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850829    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850832    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850835    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850837    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850840    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850842    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850845    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850847    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850850    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850853    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850855    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850858    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850860    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850862    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850865    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850867    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850870    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:05:41.861690 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850872    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850874    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850877    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850879    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850884    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850888    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850891    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850894    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850896    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850899    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850902    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850904    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850908    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850912    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850914    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850917    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850919    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850922    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850924    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850926    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:05:41.862151 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850929    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850931    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850934    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850936    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850939    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850942    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850944    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850947    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850949    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850952    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850954    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850956    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850959    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850961    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850964    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850966    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850970    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850973    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850976    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850978    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:05:41.862646 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850980    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850983    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850985    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850988    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850990    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850996    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.850999    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851001    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851004    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851007    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851009    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851012    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851014    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851017    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851019    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851022    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851024    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851027    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851029    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:05:41.863120 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851032    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851034    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851037    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851039    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851041    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851044    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851046    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851049    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851051    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851056    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851059    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851061    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851064    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.851066    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.858618    2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.858747    2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.858770    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.858778    2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858880    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:05:41.863561 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858888    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858893    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858898    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858902    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858906    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858911    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858915    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858919    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858923    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858927    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858932    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858936    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858940    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858944    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858948    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858952    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858956    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858959    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858962    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858965    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:05:41.864033 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858967    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858969    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858973    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858976    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858978    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858981    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858984    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858987    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858990    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858993    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858996    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.858998    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859001    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859005    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859010    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859014    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859017    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859020    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859022    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:05:41.864497 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859025    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859027    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859030    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859032    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859035    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859037    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859040    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859042    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859045    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859048    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859050    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859053    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859056    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859058    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859061    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859063    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859066    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859069    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859072    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:05:41.865052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859074    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859077    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859080    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859083    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859085    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859088    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859090    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859092    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859095    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859098    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859100    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859103    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859105    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859108    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859110    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859113    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859115    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859118    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859120    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:05:41.865526 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859122    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859133    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859136    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859138    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859141    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859144    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859146    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859149    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859151    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859154    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859156    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859159    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859162    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859164    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859167    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859170    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859172    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859175    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859178    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:05:41.866003 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859181    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:05:41.866446 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859185    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:05:41.866446 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859188    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:05:41.866446 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859192    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:05:41.866446 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859195    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:05:41.866446 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859198    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:05:41.866446 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859200    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:05:41.866446 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859203    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:05:41.866446 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859205    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:05:41.866446 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.859237    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:05:41.866446 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859374    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:05:41.866446 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859379    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:05:41.866446 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859382    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:05:41.866446 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859385    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:05:41.866446 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859388    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:05:41.866446 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859392    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:05:41.866446 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859395    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:05:41.866446 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859397    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859400    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859403    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859405    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859407    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859410    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859412    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859415    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859417    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859420    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859423    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859425    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859428    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859430    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859433    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859435    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859438    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859440    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859443    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:05:41.866885 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859446    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859448    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859452    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859456    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859459    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859462    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859465    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859468    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859471    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859473    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859475    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859478    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859481    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859483    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859486    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859488    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859491    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859493    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859497    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:05:41.867336 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859500    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859503    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859506    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859508    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859510    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859513    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859516    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859519    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859521    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859524    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859526    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859529    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859531    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859534    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859537    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859539    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859542    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859544    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859546    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859549    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:05:41.867796 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859551    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859554    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859556    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859559    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859561    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859564    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859567    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859570    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859572    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859575    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859577    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859579    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859582    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859584    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859587    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859589    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859609    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859614    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859618    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859623    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:05:41.868262 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859627    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859629    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859631    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859634    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859637    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859639    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859642    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859644    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859646    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859649    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859651    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859653    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859656    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859658    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859661    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859663    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859665    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859668    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859670    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:05:41.868750 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859674    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:05:41.869204 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859676    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:05:41.869204 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.859701    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:05:41.869204 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859733    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:05:41.869204 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859737    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:05:41.869204 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859740    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:05:41.869204 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859742    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:05:41.869204 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859745    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:05:41.869204 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859747    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:05:41.869204 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859750    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:05:41.869204 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859753    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:05:41.869204 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859755    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:05:41.869204 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859757    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:05:41.869204 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859760    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:05:41.869204 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859763    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:05:41.869204 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859765    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:05:41.869204 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859767    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859770    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859772    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859775    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859777    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859780    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859782    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859784    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859787    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859789    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859792    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859794    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859797    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859799    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859802    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859804    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859807    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859810    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859812    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:05:41.869574 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859815    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859817    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859819    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859822    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859825    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859830    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859833    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859836    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859838    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859840    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859843    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859845    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859847    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859850    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859853    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859856    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859858    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859860    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859863    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:05:41.870052 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859865    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859868    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859875    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859878    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859882    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859885    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859888    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859891    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859894    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859896    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859899    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859902    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859905    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859908    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859910    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859913    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859915    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859918    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859920    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:05:41.870492 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859922    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859925    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859927    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859929    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859932    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859934    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859937    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859939    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859942    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859945    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859947    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859949    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859952    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859954    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859957    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859960    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859963    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859965    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859968    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859971    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:05:41.870963 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859973    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859976    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859978    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859981    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859984    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859986    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859989    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859993    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859996    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.859998    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.860001    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.860003    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.860005    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.860008    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.860010    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:41.860013    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.860072    2573 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.864411    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.865636    2573 server.go:1014] "Starting client certificate rotation"
Oct 30 23:05:41.871473 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.865734    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 23:05:41.872009 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.865777    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 23:05:41.872009 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.867585    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:05:41.872009 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.868692    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:05:41.877648 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.877630    2573 log.go:25] "Validated CRI v1 runtime API"
Oct 30 23:05:41.881003 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.880988    2573 log.go:25] "Validated CRI v1 image API"
Oct 30 23:05:41.881802 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.881776    2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 23:05:41.886129 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.886097    2573 fs.go:135] Filesystem UUIDs: map[2a3a3824-3c87-4684-9503-30321fcf5616:/dev/nvme0n1p3 6197fd17-3730-4887-a18d-2d009f205b59:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 23:05:41.886206 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.886119    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 23:05:41.892587 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.892233    2573 manager.go:217] Machine: {Timestamp:2026-10-30 23:05:41.890541559 +0000 UTC m=+0.342160318 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3075476 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23ca854740cb96c3d693895bf2d508 SystemUUID:ec23ca85-4740-cb96-c3d6-93895bf2d508 BootID:c2f5b0af-ff5e-4107-8a87-a1566c9b40f4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:02:0a:8a:99:bb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:02:0a:8a:99:bb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:12:2a:c0:19:3a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 23:05:41.892713 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.892626    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 23:05:41.892786 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.892770    2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 23:05:41.895245 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.895216    2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 23:05:41.895404 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.895247    2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-205.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 23:05:41.895495 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.895409    2573 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 23:05:41.895495 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.895422    2573 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 23:05:41.895495 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.895465    2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:05:41.895495 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.895491    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:05:41.896242 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.896230    2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 23:05:41.900140 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.900124    2573 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 23:05:41.900276 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.900265    2573 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 23:05:41.900936 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.900924    2573 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 23:05:41.901000 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.900942    2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 23:05:41.901000 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.900956    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 23:05:41.901000 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.900970    2573 kubelet.go:405] "Adding apiserver pod source"
Oct 30 23:05:41.901000 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.900982    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 23:05:41.901324 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.901310    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:05:41.903398 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.903379    2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 23:05:41.906765 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.906753    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:05:41.907571 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.907558    2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 23:05:41.907630 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.907590    2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 23:05:41.907778 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.907767    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 23:05:41.907810 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.907784    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 23:05:41.907810 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.907791    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 23:05:41.907810 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.907797    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 23:05:41.907810 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.907802    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 23:05:41.907810 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.907808    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 23:05:41.908355 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.908347    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 23:05:41.908383 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.908357    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 23:05:41.908383 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.908365    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 23:05:41.908383 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.908372    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 23:05:41.908456 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.908387    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 23:05:41.908456 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.908396    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 23:05:41.909054 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.909045    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 23:05:41.909054 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.909055    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 23:05:41.912772 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.912758    2573 server.go:1263] "Started kubelet"
Oct 30 23:05:41.912919 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.912877    2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 23:05:41.913074 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.912873    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 23:05:41.913131 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.913095    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 23:05:41.913433 ip-10-0-130-205 systemd[1]: Started Kubernetes Kubelet.
Oct 30 23:05:41.914215 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.914066    2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 23:05:41.914988 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.914971    2573 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 23:05:41.918625 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:41.918582    2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 23:05:41.921173 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.921152    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 23:05:41.921463 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.921445    2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 23:05:41.922045 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:41.921569    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-205.us-west-1.compute.internal\" not found"
Oct 30 23:05:41.922287 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.922271    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 23:05:41.922337 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.922272    2573 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 23:05:41.922337 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.922300    2573 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 23:05:41.922403 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.922344    2573 factory.go:55] Registering systemd factory
Oct 30 23:05:41.922403 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.922362    2573 factory.go:223] Registration of the systemd container factory successfully
Oct 30 23:05:41.922475 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.922349    2573 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 23:05:41.922475 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.922417    2573 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 23:05:41.922573 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.922562    2573 factory.go:153] Registering CRI-O factory
Oct 30 23:05:41.922626 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.922576    2573 factory.go:223] Registration of the crio container factory successfully
Oct 30 23:05:41.922657 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.922642    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 23:05:41.922657 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.922656    2573 factory.go:103] Registering Raw factory
Oct 30 23:05:41.922723 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.922666    2573 manager.go:1199] Started watching for new ooms in manager
Oct 30 23:05:41.923405 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.923387    2573 manager.go:319] Starting recovery of all containers
Oct 30 23:05:41.934470 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.934452    2573 manager.go:324] Recovery completed
Oct 30 23:05:41.938687 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.938673    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:05:41.940235 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.940221    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:05:41.940282 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.940248    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:05:41.940282 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.940258    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:05:41.940796 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.940781    2573 cpu_manager.go:226] "Starting" policy="none"
Oct 30 23:05:41.940850 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.940797    2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 23:05:41.940850 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.940806    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:05:41.940850 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.940816    2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 23:05:41.941442 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.941424    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:05:41.941498 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.941451    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:05:41.941498 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.941461    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:05:41.942900 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.942888    2573 policy_none.go:50] "Start"
Oct 30 23:05:41.942960 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.942904    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 23:05:41.942960 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.942914    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 23:05:41.944849 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.944839    2573 policy_none.go:44] "Start"
Oct 30 23:05:42.009913 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.991176    2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 23:05:42.009913 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.994140    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 23:05:42.009913 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.995418    2573 manager.go:341] "Starting Device Plugin manager"
Oct 30 23:05:42.009913 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:41.995451    2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 23:05:42.009913 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.995462    2573 server.go:87] "Starting device plugin registration server"
Oct 30 23:05:42.009913 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.995465    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 23:05:42.009913 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.995478    2573 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 23:05:42.009913 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.995498    2573 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 23:05:42.009913 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:41.995536    2573 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 23:05:42.009913 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.995740    2573 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 23:05:42.009913 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.995752    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 23:05:42.009913 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.995910    2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 23:05:42.009913 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.995999    2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 23:05:42.009913 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:41.996019    2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 23:05:42.009913 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:41.996302    2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 23:05:42.009913 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:41.996351    2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-205.us-west-1.compute.internal\" not found"
Oct 30 23:05:42.056927 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.056906    2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-205.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 23:05:42.056927 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:42.056916    2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-205.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 23:05:42.058091 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:42.056929    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-205.us-west-1.compute.internal.18e371335558a6e6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-205.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-205.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 23:05:41.912733414 +0000 UTC m=+0.364352172,LastTimestamp:2026-10-30 23:05:41.912733414 +0000 UTC m=+0.364352172,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-205.us-west-1.compute.internal,}"
Oct 30 23:05:42.096276 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.096213    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:05:42.096276 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.096226    2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal"]
Oct 30 23:05:42.096415 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.096288    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:05:42.097211 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.097193    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:05:42.097297 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.097223    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:05:42.097297 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.097230    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:05:42.097297 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.097250    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:05:42.097297 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.097267    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:05:42.097297 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.097267    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:05:42.097485 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.097336    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.099553 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.099540    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:05:42.099695 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.099681    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.099746 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.099722    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:05:42.100368 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.100354    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:05:42.100444 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.100375    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:05:42.100444 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.100384    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:05:42.100444 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.100423    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:05:42.100444 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.100442    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:05:42.100575 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.100453    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:05:42.102582 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.102568    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.102660 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.102611    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:05:42.103350 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.103336    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:05:42.103418 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.103357    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:05:42.103418 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.103376    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:05:42.123856 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.123835    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bc6014254655639fce61607de5fe088-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal\" (UID: \"8bc6014254655639fce61607de5fe088\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.123954 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.123868    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bc6014254655639fce61607de5fe088-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal\" (UID: \"8bc6014254655639fce61607de5fe088\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.123954 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.123887    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1b66be07f6b0e09e908458ab3790deb-config\") pod \"kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal\" (UID: \"a1b66be07f6b0e09e908458ab3790deb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.128935 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:42.128911    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-205.us-west-1.compute.internal\" not found" node="ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.131461 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:42.131445    2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-205.us-west-1.compute.internal\" not found" node="ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.138068 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.138049    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 23:05:42.138154 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.138097    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7mpwq"
Oct 30 23:05:42.152948 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:42.152922    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 23:05:42.152948 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:42.152925    2573 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 23:05:42.153126 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:42.152881    2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-205.us-west-1.compute.internal.18e3713356fc4cbb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-205.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-205.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-205.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 23:05:41.940235451 +0000 UTC m=+0.391854209,LastTimestamp:2026-10-30 23:05:41.940235451 +0000 UTC m=+0.391854209,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-205.us-west-1.compute.internal,}"
Oct 30 23:05:42.211777 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.211751    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:05:42.212623 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:42.212608    2573 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-130-205.us-west-1.compute.internal\" not found" node="ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.214123 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.214108    2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:05:42.224663 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.224639    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bc6014254655639fce61607de5fe088-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal\" (UID: \"8bc6014254655639fce61607de5fe088\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.224758 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.224670    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bc6014254655639fce61607de5fe088-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal\" (UID: \"8bc6014254655639fce61607de5fe088\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.224758 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.224695    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1b66be07f6b0e09e908458ab3790deb-config\") pod \"kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal\" (UID: \"a1b66be07f6b0e09e908458ab3790deb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.224758 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.224740    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bc6014254655639fce61607de5fe088-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal\" (UID: \"8bc6014254655639fce61607de5fe088\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.224758 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.224748    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1b66be07f6b0e09e908458ab3790deb-config\") pod \"kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal\" (UID: \"a1b66be07f6b0e09e908458ab3790deb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.224897 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.224748    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bc6014254655639fce61607de5fe088-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal\" (UID: \"8bc6014254655639fce61607de5fe088\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.361739 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:42.361667    2573 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-130-205.us-west-1.compute.internal\" not found" node="ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.412869 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.412849    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:05:42.413760 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.413743    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:05:42.413853 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.413774    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:05:42.413853 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.413785    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:05:42.413853 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.413820    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.430931 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.430909    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.434065 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.434052    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.471067 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.471045    2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.521834 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.521811    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.596591 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.596565    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:05:42.597640 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.597625    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:05:42.657065 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.656994    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:05:42.865415 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.865383    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 23:05:42.865931 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.865542    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 23:05:42.865931 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:42.865522    2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a69c238be523043a7aa0939d4b6f2b19-5c317881c9fa0372.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-130-205.us-west-1.compute.internal.18e3713356fcb532\": read tcp 10.0.130.205:58870->3.16.240.75:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-130-205.us-west-1.compute.internal.18e3713356fcb532  default   4799 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-205.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-130-205.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-130-205.us-west-1.compute.internal,},FirstTimestamp:2026-10-30 23:05:41 +0000 UTC,LastTimestamp:2026-10-30 23:05:42.097271913 +0000 UTC m=+0.548890672,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-205.us-west-1.compute.internal,}"
Oct 30 23:05:42.901733 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.901702    2573 apiserver.go:52] "Watching apiserver"
Oct 30 23:05:42.921638 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:42.921582    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 23:05:43.142305 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.142265    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:00:42 +0000 UTC" deadline="2028-08-11 16:39:29.566088322 +0000 UTC"
Oct 30 23:05:43.142305 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.142294    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15617h33m46.423798337s"
Oct 30 23:05:43.182203 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.182133    2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 23:05:43.182203 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.182165    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:05:43.182530 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.182508    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-qvbgk","kube-system/global-pull-secret-syncer-cthfm","openshift-dns/node-resolver-gldcx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-pln6h","openshift-network-operator/iptables-alerter-dzklv","kube-system/konnectivity-agent-fg8fg","openshift-image-registry/node-ca-2g2gr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79","openshift-cluster-node-tuning-operator/tuned-rcgrg","openshift-ovn-kubernetes/ovnkube-node-4c5rm","kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-z7d8k","openshift-multus/multus-wk66h"]
Oct 30 23:05:43.185012 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.184992    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:05:43.185085 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.185058    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvbgk" podUID="70a51653-2167-4e11-bf04-81f5b10c9c9d"
Oct 30 23:05:43.187027 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.187010    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:43.187112 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.187055    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cthfm" podUID="2e942185-68e3-4d57-9f42-49d2965b41e4"
Oct 30 23:05:43.189065 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.189045    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gldcx"
Oct 30 23:05:43.191093 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.191072    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:43.191185 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.191134    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pln6h" podUID="58ec1adc-599b-4796-b12e-1cc85a834b15"
Oct 30 23:05:43.193102 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.193085    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dzklv"
Oct 30 23:05:43.195202 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.195186    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fg8fg"
Oct 30 23:05:43.197273 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.197256    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2g2gr"
Oct 30 23:05:43.199355 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.199340    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.201609 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.201577    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.203790 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.203775    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.206181 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.206160    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.208488 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.208462    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.224631 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.224613    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 23:05:43.230192 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230174    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-sysctl-conf\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.230275 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230203    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-cni-bin\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.230275 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230226    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.230275 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230251    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-cnibin\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.230275 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230273    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-lib-modules\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.230476 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230312    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-var-lib-kubelet\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.230476 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230335    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-system-cni-dir\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.230476 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230358    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-etc-kubernetes\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.230476 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230381    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q46ww\" (UniqueName: \"kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww\") pod \"network-check-target-qvbgk\" (UID: \"70a51653-2167-4e11-bf04-81f5b10c9c9d\") " pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:05:43.230476 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230405    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/108c7917-29b6-47bc-a590-35fd94c3de19-tmp-dir\") pod \"node-resolver-gldcx\" (UID: \"108c7917-29b6-47bc-a590-35fd94c3de19\") " pod="openshift-dns/node-resolver-gldcx"
Oct 30 23:05:43.230476 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230428    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf-host-slash\") pod \"iptables-alerter-dzklv\" (UID: \"29261ce8-6b07-4ea5-afa4-8500cc1c7bbf\") " pod="openshift-network-operator/iptables-alerter-dzklv"
Oct 30 23:05:43.230813 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230498    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-modprobe-d\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.230813 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230529    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-run\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.230813 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230546    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-run-systemd\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.230813 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230611    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-run-ovn-kubernetes\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.230813 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230639    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-ovn-node-metrics-cert\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.230813 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230663    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-multus-conf-dir\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.230813 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230700    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6bjr\" (UniqueName: \"kubernetes.io/projected/81b805c0-763e-4cd9-b6f5-af98767b298e-kube-api-access-d6bjr\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.230813 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230745    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5p98\" (UniqueName: \"kubernetes.io/projected/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-kube-api-access-z5p98\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.230813 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230772    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e942185-68e3-4d57-9f42-49d2965b41e4-dbus\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:43.230813 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230788    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs\") pod \"network-metrics-daemon-pln6h\" (UID: \"58ec1adc-599b-4796-b12e-1cc85a834b15\") " pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:43.230813 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230807    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-slash\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.231211 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230829    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.231211 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230856    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-env-overrides\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.231211 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230878    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.231211 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230902    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-multus-cni-dir\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.231211 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230927    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-var-lib-kubelet\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.231211 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230946    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-hostroot\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.231211 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.230994    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df39047e-a25e-4661-a34e-a6bca0f344f2-host\") pod \"node-ca-2g2gr\" (UID: \"df39047e-a25e-4661-a34e-a6bca0f344f2\") " pod="openshift-image-registry/node-ca-2g2gr"
Oct 30 23:05:43.231211 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231033    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-registration-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.231211 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231060    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-systemd\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.231211 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231083    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-ovnkube-script-lib\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.231211 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231106    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-run-multus-certs\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.231211 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231129    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-sysconfig\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.231211 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231149    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-sys\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.231211 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231171    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-var-lib-openvswitch\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.231211 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231193    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-run-ovn\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.231639 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231227    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqblk\" (UniqueName: \"kubernetes.io/projected/91deced5-f22e-4148-98da-6a232be165c3-kube-api-access-kqblk\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.231639 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231246    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7e14c04d-b583-4f0a-840c-d493b2a2bbaf-agent-certs\") pod \"konnectivity-agent-fg8fg\" (UID: \"7e14c04d-b583-4f0a-840c-d493b2a2bbaf\") " pod="kube-system/konnectivity-agent-fg8fg"
Oct 30 23:05:43.231639 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231262    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7e14c04d-b583-4f0a-840c-d493b2a2bbaf-konnectivity-ca\") pod \"konnectivity-agent-fg8fg\" (UID: \"7e14c04d-b583-4f0a-840c-d493b2a2bbaf\") " pod="kube-system/konnectivity-agent-fg8fg"
Oct 30 23:05:43.231639 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231275    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-socket-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.231639 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231288    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-sys-fs\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.231639 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231307    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-kubelet\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.231639 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231327    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-etc-openvswitch\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.231639 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231340    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91deced5-f22e-4148-98da-6a232be165c3-multus-daemon-config\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.231639 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231360    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e942185-68e3-4d57-9f42-49d2965b41e4-kubelet-config\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:43.231639 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231375    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-systemd-units\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.231639 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231412    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq95j\" (UniqueName: \"kubernetes.io/projected/58ec1adc-599b-4796-b12e-1cc85a834b15-kube-api-access-mq95j\") pod \"network-metrics-daemon-pln6h\" (UID: \"58ec1adc-599b-4796-b12e-1cc85a834b15\") " pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:43.231639 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231430    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mdsg\" (UniqueName: \"kubernetes.io/projected/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf-kube-api-access-9mdsg\") pod \"iptables-alerter-dzklv\" (UID: \"29261ce8-6b07-4ea5-afa4-8500cc1c7bbf\") " pod="openshift-network-operator/iptables-alerter-dzklv"
Oct 30 23:05:43.231639 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231457    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmhb8\" (UniqueName: \"kubernetes.io/projected/1918245f-b465-485f-9603-04ebe4846085-kube-api-access-pmhb8\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.231639 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231501    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-host\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.231639 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231520    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-ovnkube-config\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.231639 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231552    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-cnibin\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.232267 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231567    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.232267 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231581    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-multus-socket-dir-parent\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.232267 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231611    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-system-cni-dir\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.232267 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231650    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-run-netns\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.232267 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231670    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-var-lib-cni-bin\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.232267 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231690    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:43.232267 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231706    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/108c7917-29b6-47bc-a590-35fd94c3de19-hosts-file\") pod \"node-resolver-gldcx\" (UID: \"108c7917-29b6-47bc-a590-35fd94c3de19\") " pod="openshift-dns/node-resolver-gldcx"
Oct 30 23:05:43.232267 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231733    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqgxb\" (UniqueName: \"kubernetes.io/projected/108c7917-29b6-47bc-a590-35fd94c3de19-kube-api-access-rqgxb\") pod \"node-resolver-gldcx\" (UID: \"108c7917-29b6-47bc-a590-35fd94c3de19\") " pod="openshift-dns/node-resolver-gldcx"
Oct 30 23:05:43.232267 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231766    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df39047e-a25e-4661-a34e-a6bca0f344f2-serviceca\") pod \"node-ca-2g2gr\" (UID: \"df39047e-a25e-4661-a34e-a6bca0f344f2\") " pod="openshift-image-registry/node-ca-2g2gr"
Oct 30 23:05:43.232267 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231797    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-device-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.232267 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231820    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf-iptables-alerter-script\") pod \"iptables-alerter-dzklv\" (UID: \"29261ce8-6b07-4ea5-afa4-8500cc1c7bbf\") " pod="openshift-network-operator/iptables-alerter-dzklv"
Oct 30 23:05:43.232267 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231838    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-tuned\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.232267 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231869    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-node-log\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.232267 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231895    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb2fp\" (UniqueName: \"kubernetes.io/projected/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-kube-api-access-pb2fp\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.232267 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231952    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-sysctl-d\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.232267 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.231978    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-run-netns\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.232728 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.232005    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-run-openvswitch\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.232728 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.232026    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-cni-netd\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.232728 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.232043    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91deced5-f22e-4148-98da-6a232be165c3-cni-binary-copy\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.232728 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.232066    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-run-k8s-cni-cncf-io\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.232728 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.232089    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.232728 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.232110    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-kubernetes\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.232728 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.232139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81b805c0-763e-4cd9-b6f5-af98767b298e-tmp\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.232728 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.232170    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-log-socket\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.232728 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.232197    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.232728 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.232222    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-var-lib-cni-multus\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.232728 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.232244    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5zjd\" (UniqueName: \"kubernetes.io/projected/df39047e-a25e-4661-a34e-a6bca0f344f2-kube-api-access-p5zjd\") pod \"node-ca-2g2gr\" (UID: \"df39047e-a25e-4661-a34e-a6bca0f344f2\") " pod="openshift-image-registry/node-ca-2g2gr"
Oct 30 23:05:43.232728 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.232288    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-etc-selinux\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.250126 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.250109    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4qgbv\""
Oct 30 23:05:43.250126 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.250115    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4j2qm\""
Oct 30 23:05:43.250272 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.250170    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ldcvv\""
Oct 30 23:05:43.250272 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.250211    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 23:05:43.250272 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.250226    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:05:43.250272 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.250249    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 23:05:43.250272 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.250273    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 23:05:43.252331 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.252316    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 23:05:43.252447 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.252432    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 23:05:43.252509 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.252439    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 23:05:43.252683 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.252668    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 23:05:43.253014 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.253000    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 23:05:43.253141 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.253122    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 23:05:43.253141 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.253127    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 23:05:43.253436 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.253421    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 23:05:43.255586 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.255571    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:05:43.255768 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.255755    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 23:05:43.256958 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.256942    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 23:05:43.257030 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.257006    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nlg56\""
Oct 30 23:05:43.258741 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.258723    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-46zpf\""
Oct 30 23:05:43.258741 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.258725    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9xqsx\""
Oct 30 23:05:43.258866 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.258776    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 23:05:43.285390 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.285372    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 23:05:43.300878 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.300863    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 23:05:43.300965 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.300897    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 23:05:43.300965 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.300913    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 23:05:43.301054 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.300976    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t7twr\""
Oct 30 23:05:43.301054 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.301007    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 23:05:43.326258 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.326241    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 23:05:43.326359 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.326301    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-djrd8"
Oct 30 23:05:43.332875 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.332858    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-sysctl-conf\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.332968 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.332881    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-cni-bin\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.332968 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.332896    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.332968 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.332910    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-cnibin\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.332968 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.332929    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-lib-modules\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.332968 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.332950    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-var-lib-kubelet\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.333139 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.332974    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-cni-bin\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.333139 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333004    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-var-lib-kubelet\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.333139 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333011    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-cnibin\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.333139 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333052    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-sysctl-conf\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.333139 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333049    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-system-cni-dir\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.333139 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333066    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-lib-modules\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.333139 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333073    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-system-cni-dir\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.333139 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333082    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-etc-kubernetes\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.333139 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333105    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-etc-kubernetes\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.333139 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333108    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46ww\" (UniqueName: \"kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww\") pod \"network-check-target-qvbgk\" (UID: \"70a51653-2167-4e11-bf04-81f5b10c9c9d\") " pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:05:43.333139 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333131    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/108c7917-29b6-47bc-a590-35fd94c3de19-tmp-dir\") pod \"node-resolver-gldcx\" (UID: \"108c7917-29b6-47bc-a590-35fd94c3de19\") " pod="openshift-dns/node-resolver-gldcx"
Oct 30 23:05:43.333516 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333145    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf-host-slash\") pod \"iptables-alerter-dzklv\" (UID: \"29261ce8-6b07-4ea5-afa4-8500cc1c7bbf\") " pod="openshift-network-operator/iptables-alerter-dzklv"
Oct 30 23:05:43.333516 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333183    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf-host-slash\") pod \"iptables-alerter-dzklv\" (UID: \"29261ce8-6b07-4ea5-afa4-8500cc1c7bbf\") " pod="openshift-network-operator/iptables-alerter-dzklv"
Oct 30 23:05:43.333516 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333238    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-modprobe-d\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.333516 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333259    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-run\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.333516 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333275    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-run-systemd\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.333516 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333289    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-run-ovn-kubernetes\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.333516 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333325    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-ovn-node-metrics-cert\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.333516 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333330    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-modprobe-d\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.333516 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333343    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-multus-conf-dir\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.333516 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333375    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-run-ovn-kubernetes\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.333516 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333380    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-multus-conf-dir\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.333516 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6bjr\" (UniqueName: \"kubernetes.io/projected/81b805c0-763e-4cd9-b6f5-af98767b298e-kube-api-access-d6bjr\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.333516 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333410    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-run-systemd\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.333516 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333418    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-run\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.333516 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333427    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5p98\" (UniqueName: \"kubernetes.io/projected/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-kube-api-access-z5p98\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.333516 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333429    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/108c7917-29b6-47bc-a590-35fd94c3de19-tmp-dir\") pod \"node-resolver-gldcx\" (UID: \"108c7917-29b6-47bc-a590-35fd94c3de19\") " pod="openshift-dns/node-resolver-gldcx"
Oct 30 23:05:43.333516 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333448    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e942185-68e3-4d57-9f42-49d2965b41e4-dbus\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:43.333516 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333475    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs\") pod \"network-metrics-daemon-pln6h\" (UID: \"58ec1adc-599b-4796-b12e-1cc85a834b15\") " pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:43.334394 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333499    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-slash\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.334394 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333521    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.334394 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333548    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-env-overrides\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.334394 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333566    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-slash\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.334394 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333570    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.334394 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333524    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e942185-68e3-4d57-9f42-49d2965b41e4-dbus\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:43.334394 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333622    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-multus-cni-dir\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.334394 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333651    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-var-lib-kubelet\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.334394 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333668    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.334394 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333676    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-hostroot\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.334394 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333693    2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 23:05:43.334394 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333717    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df39047e-a25e-4661-a34e-a6bca0f344f2-host\") pod \"node-ca-2g2gr\" (UID: \"df39047e-a25e-4661-a34e-a6bca0f344f2\") " pod="openshift-image-registry/node-ca-2g2gr"
Oct 30 23:05:43.334394 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333743    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-var-lib-kubelet\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.334394 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333753    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-multus-cni-dir\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.334394 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333763    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-hostroot\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.334394 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333698    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df39047e-a25e-4661-a34e-a6bca0f344f2-host\") pod \"node-ca-2g2gr\" (UID: \"df39047e-a25e-4661-a34e-a6bca0f344f2\") " pod="openshift-image-registry/node-ca-2g2gr"
Oct 30 23:05:43.334394 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333802    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-registration-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.334394 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333835    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-systemd\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.335188 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333858    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-ovnkube-script-lib\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.335188 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333882    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-registration-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.335188 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333882    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-run-multus-certs\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.335188 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333901    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-run-multus-certs\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.335188 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333920    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-sysconfig\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.335188 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333930    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-systemd\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.335188 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333934    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-sys\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.335188 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333949    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-var-lib-openvswitch\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.335188 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333964    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-run-ovn\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.335188 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.333997    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqblk\" (UniqueName: \"kubernetes.io/projected/91deced5-f22e-4148-98da-6a232be165c3-kube-api-access-kqblk\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.335188 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334012    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7e14c04d-b583-4f0a-840c-d493b2a2bbaf-agent-certs\") pod \"konnectivity-agent-fg8fg\" (UID: \"7e14c04d-b583-4f0a-840c-d493b2a2bbaf\") " pod="kube-system/konnectivity-agent-fg8fg"
Oct 30 23:05:43.335188 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.334028    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:05:43.335188 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.334100    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs podName:58ec1adc-599b-4796-b12e-1cc85a834b15 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:43.834082742 +0000 UTC m=+2.285701520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs") pod "network-metrics-daemon-pln6h" (UID: "58ec1adc-599b-4796-b12e-1cc85a834b15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:05:43.335188 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334028    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7e14c04d-b583-4f0a-840c-d493b2a2bbaf-konnectivity-ca\") pod \"konnectivity-agent-fg8fg\" (UID: \"7e14c04d-b583-4f0a-840c-d493b2a2bbaf\") " pod="kube-system/konnectivity-agent-fg8fg"
Oct 30 23:05:43.335188 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334136    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-socket-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.335188 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334150    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-var-lib-openvswitch\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.335188 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334159    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-sys-fs\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.335922 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334162    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-env-overrides\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.335922 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334192    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-sys-fs\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.335922 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334211    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-sysconfig\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.335922 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334236    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-kubelet\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.335922 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334246    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-sys\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.335922 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334266    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-etc-openvswitch\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.335922 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334270    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-socket-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.335922 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334277    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-kubelet\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.335922 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334310    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91deced5-f22e-4148-98da-6a232be165c3-multus-daemon-config\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.335922 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334330    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-etc-openvswitch\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.335922 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334335    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e942185-68e3-4d57-9f42-49d2965b41e4-kubelet-config\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:43.335922 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334360    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-systemd-units\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.335922 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334384    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq95j\" (UniqueName: \"kubernetes.io/projected/58ec1adc-599b-4796-b12e-1cc85a834b15-kube-api-access-mq95j\") pod \"network-metrics-daemon-pln6h\" (UID: \"58ec1adc-599b-4796-b12e-1cc85a834b15\") " pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:43.335922 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334395    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7e14c04d-b583-4f0a-840c-d493b2a2bbaf-konnectivity-ca\") pod \"konnectivity-agent-fg8fg\" (UID: \"7e14c04d-b583-4f0a-840c-d493b2a2bbaf\") " pod="kube-system/konnectivity-agent-fg8fg"
Oct 30 23:05:43.335922 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334410    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mdsg\" (UniqueName: \"kubernetes.io/projected/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf-kube-api-access-9mdsg\") pod \"iptables-alerter-dzklv\" (UID: \"29261ce8-6b07-4ea5-afa4-8500cc1c7bbf\") " pod="openshift-network-operator/iptables-alerter-dzklv"
Oct 30 23:05:43.335922 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334447    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-run-ovn\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.335922 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334463    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-ovnkube-script-lib\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.336946 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334477    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-systemd-units\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.336946 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334493    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e942185-68e3-4d57-9f42-49d2965b41e4-kubelet-config\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:43.336946 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334527    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmhb8\" (UniqueName: \"kubernetes.io/projected/1918245f-b465-485f-9603-04ebe4846085-kube-api-access-pmhb8\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.336946 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334553    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-host\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.336946 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334590    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-host\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.336946 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334682    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-ovnkube-config\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.336946 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334706    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-cnibin\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.336946 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334722    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.336946 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334736    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-multus-socket-dir-parent\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.336946 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334751    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-system-cni-dir\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.336946 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334772    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-run-netns\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.336946 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334776    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-cnibin\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.336946 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334786    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-var-lib-cni-bin\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.336946 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334812    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-var-lib-cni-bin\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.336946 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334818    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-multus-socket-dir-parent\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.336946 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334862    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-system-cni-dir\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.336946 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334893    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.336946 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334957    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-run-netns\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.337833 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.334976    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:43.337833 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335002    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/108c7917-29b6-47bc-a590-35fd94c3de19-hosts-file\") pod \"node-resolver-gldcx\" (UID: \"108c7917-29b6-47bc-a590-35fd94c3de19\") " pod="openshift-dns/node-resolver-gldcx"
Oct 30 23:05:43.337833 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335025    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqgxb\" (UniqueName: \"kubernetes.io/projected/108c7917-29b6-47bc-a590-35fd94c3de19-kube-api-access-rqgxb\") pod \"node-resolver-gldcx\" (UID: \"108c7917-29b6-47bc-a590-35fd94c3de19\") " pod="openshift-dns/node-resolver-gldcx"
Oct 30 23:05:43.337833 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335065    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df39047e-a25e-4661-a34e-a6bca0f344f2-serviceca\") pod \"node-ca-2g2gr\" (UID: \"df39047e-a25e-4661-a34e-a6bca0f344f2\") " pod="openshift-image-registry/node-ca-2g2gr"
Oct 30 23:05:43.337833 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335087    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-device-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.337833 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335111    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf-iptables-alerter-script\") pod \"iptables-alerter-dzklv\" (UID: \"29261ce8-6b07-4ea5-afa4-8500cc1c7bbf\") " pod="openshift-network-operator/iptables-alerter-dzklv"
Oct 30 23:05:43.337833 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335131    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-tuned\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.337833 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335152    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-node-log\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.337833 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.335164    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:05:43.337833 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335174    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb2fp\" (UniqueName: \"kubernetes.io/projected/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-kube-api-access-pb2fp\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.337833 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335196    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-sysctl-d\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.337833 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.335209    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret podName:2e942185-68e3-4d57-9f42-49d2965b41e4 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:43.83519456 +0000 UTC m=+2.286813322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret") pod "global-pull-secret-syncer-cthfm" (UID: "2e942185-68e3-4d57-9f42-49d2965b41e4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:05:43.337833 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335223    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/108c7917-29b6-47bc-a590-35fd94c3de19-hosts-file\") pod \"node-resolver-gldcx\" (UID: \"108c7917-29b6-47bc-a590-35fd94c3de19\") " pod="openshift-dns/node-resolver-gldcx"
Oct 30 23:05:43.337833 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335236    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-run-netns\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.337833 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335262    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-run-openvswitch\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.337833 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335271    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-sysctl-d\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.337833 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335274    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-ovnkube-config\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.338624 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335285    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-cni-netd\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.338624 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335309    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91deced5-f22e-4148-98da-6a232be165c3-cni-binary-copy\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.338624 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335334    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-run-k8s-cni-cncf-io\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.338624 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335362    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.338624 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335387    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-kubernetes\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.338624 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335412    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81b805c0-763e-4cd9-b6f5-af98767b298e-tmp\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.338624 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335446    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-log-socket\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.338624 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335470    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-device-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.338624 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335492    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.338624 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335521    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-node-log\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.338624 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335862    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-run-k8s-cni-cncf-io\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.338624 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335917    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-run-openvswitch\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.338624 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335965    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df39047e-a25e-4661-a34e-a6bca0f344f2-serviceca\") pod \"node-ca-2g2gr\" (UID: \"df39047e-a25e-4661-a34e-a6bca0f344f2\") " pod="openshift-image-registry/node-ca-2g2gr"
Oct 30 23:05:43.338624 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335960    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-run-netns\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.338624 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.336020    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-cni-netd\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.338624 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.336084    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-log-socket\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.338624 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.336168    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.339335 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.335521    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-var-lib-cni-multus\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.339335 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.336229    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5zjd\" (UniqueName: \"kubernetes.io/projected/df39047e-a25e-4661-a34e-a6bca0f344f2-kube-api-access-p5zjd\") pod \"node-ca-2g2gr\" (UID: \"df39047e-a25e-4661-a34e-a6bca0f344f2\") " pod="openshift-image-registry/node-ca-2g2gr"
Oct 30 23:05:43.339335 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.336255    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-etc-selinux\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.339335 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.336509    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-etc-selinux\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.339335 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.336764    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf-iptables-alerter-script\") pod \"iptables-alerter-dzklv\" (UID: \"29261ce8-6b07-4ea5-afa4-8500cc1c7bbf\") " pod="openshift-network-operator/iptables-alerter-dzklv"
Oct 30 23:05:43.339335 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.336845    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-kubernetes\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.339335 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.337829    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-ovn-node-metrics-cert\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.339335 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.337945    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7e14c04d-b583-4f0a-840c-d493b2a2bbaf-agent-certs\") pod \"konnectivity-agent-fg8fg\" (UID: \"7e14c04d-b583-4f0a-840c-d493b2a2bbaf\") " pod="kube-system/konnectivity-agent-fg8fg"
Oct 30 23:05:43.339335 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.338408    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-tuned\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.339335 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.338487    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81b805c0-763e-4cd9-b6f5-af98767b298e-tmp\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.340117 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.340092    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-var-lib-cni-multus\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.347125 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.347105    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 23:05:43.347125 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.347117    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 23:05:43.347230 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.347158    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 23:05:43.347230 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.347164    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 23:05:43.347230 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.347171    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 23:05:43.347230 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.347212    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 23:05:43.348818 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.348802    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7m4ss\""
Oct 30 23:05:43.348878 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.348844    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7sr97\""
Oct 30 23:05:43.354043 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.354022    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.354104 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.354053    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.355274 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.355258    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91deced5-f22e-4148-98da-6a232be165c3-multus-daemon-config\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.357478 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.357462    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.357519 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.357486    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91deced5-f22e-4148-98da-6a232be165c3-cni-binary-copy\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.389955 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.389937    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:05:43.390151 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.389958    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:05:43.390151 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.389972    2573 projected.go:196] Error preparing data for projected volume kube-api-access-q46ww for pod openshift-network-diagnostics/network-check-target-qvbgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:05:43.390151 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.390028    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww podName:70a51653-2167-4e11-bf04-81f5b10c9c9d nodeName:}" failed. No retries permitted until 2026-10-30 23:05:43.890011396 +0000 UTC m=+2.341630156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-q46ww" (UniqueName: "kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww") pod "network-check-target-qvbgk" (UID: "70a51653-2167-4e11-bf04-81f5b10c9c9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:05:43.392563 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.392489    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6bjr\" (UniqueName: \"kubernetes.io/projected/81b805c0-763e-4cd9-b6f5-af98767b298e-kube-api-access-d6bjr\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.392806 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.392774    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5p98\" (UniqueName: \"kubernetes.io/projected/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-kube-api-access-z5p98\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.392911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.392832    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmhb8\" (UniqueName: \"kubernetes.io/projected/1918245f-b465-485f-9603-04ebe4846085-kube-api-access-pmhb8\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.392911 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.392864    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqblk\" (UniqueName: \"kubernetes.io/projected/91deced5-f22e-4148-98da-6a232be165c3-kube-api-access-kqblk\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.393286 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.393266    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mdsg\" (UniqueName: \"kubernetes.io/projected/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf-kube-api-access-9mdsg\") pod \"iptables-alerter-dzklv\" (UID: \"29261ce8-6b07-4ea5-afa4-8500cc1c7bbf\") " pod="openshift-network-operator/iptables-alerter-dzklv"
Oct 30 23:05:43.393369 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.393293    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq95j\" (UniqueName: \"kubernetes.io/projected/58ec1adc-599b-4796-b12e-1cc85a834b15-kube-api-access-mq95j\") pod \"network-metrics-daemon-pln6h\" (UID: \"58ec1adc-599b-4796-b12e-1cc85a834b15\") " pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:43.393449 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.393429    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqgxb\" (UniqueName: \"kubernetes.io/projected/108c7917-29b6-47bc-a590-35fd94c3de19-kube-api-access-rqgxb\") pod \"node-resolver-gldcx\" (UID: \"108c7917-29b6-47bc-a590-35fd94c3de19\") " pod="openshift-dns/node-resolver-gldcx"
Oct 30 23:05:43.394851 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.394830    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb2fp\" (UniqueName: \"kubernetes.io/projected/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-kube-api-access-pb2fp\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.395358 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.395339    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5zjd\" (UniqueName: \"kubernetes.io/projected/df39047e-a25e-4661-a34e-a6bca0f344f2-kube-api-access-p5zjd\") pod \"node-ca-2g2gr\" (UID: \"df39047e-a25e-4661-a34e-a6bca0f344f2\") " pod="openshift-image-registry/node-ca-2g2gr"
Oct 30 23:05:43.461490 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.461437    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:05:43.524656 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.524622    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gldcx"
Oct 30 23:05:43.536385 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.536362    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dzklv"
Oct 30 23:05:43.542583 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.542563    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-fg8fg"
Oct 30 23:05:43.559581 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.559558    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2g2gr"
Oct 30 23:05:43.576537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.576516    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:05:43.578336 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.578318    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:05:43.596391 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:43.596361    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b66be07f6b0e09e908458ab3790deb.slice/crio-9f28f4ffabe36c7eb7853ab824409f4c93fc666101154ea2a9de302865ebb19a WatchSource:0}: Error finding container 9f28f4ffabe36c7eb7853ab824409f4c93fc666101154ea2a9de302865ebb19a: Status 404 returned error can't find the container with id 9f28f4ffabe36c7eb7853ab824409f4c93fc666101154ea2a9de302865ebb19a
Oct 30 23:05:43.596816 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:43.596787    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf39047e_a25e_4661_a34e_a6bca0f344f2.slice/crio-8d753418031592ca311fa0bc6cb8bcc039543c68e5ec696435c1dcbfc260c320 WatchSource:0}: Error finding container 8d753418031592ca311fa0bc6cb8bcc039543c68e5ec696435c1dcbfc260c320: Status 404 returned error can't find the container with id 8d753418031592ca311fa0bc6cb8bcc039543c68e5ec696435c1dcbfc260c320
Oct 30 23:05:43.597373 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:43.597353    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bc6014254655639fce61607de5fe088.slice/crio-0c0ebf017ce4c7526dddc3e8f9d6f9518b84b7ca69a5085f1e40912bbfd5fc03 WatchSource:0}: Error finding container 0c0ebf017ce4c7526dddc3e8f9d6f9518b84b7ca69a5085f1e40912bbfd5fc03: Status 404 returned error can't find the container with id 0c0ebf017ce4c7526dddc3e8f9d6f9518b84b7ca69a5085f1e40912bbfd5fc03
Oct 30 23:05:43.598331 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:43.598312    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1918245f_b465_485f_9603_04ebe4846085.slice/crio-e8d2abb44c70942aa77687775e2477d94c0dd840609a45ec23e29865c86c6d15 WatchSource:0}: Error finding container e8d2abb44c70942aa77687775e2477d94c0dd840609a45ec23e29865c86c6d15: Status 404 returned error can't find the container with id e8d2abb44c70942aa77687775e2477d94c0dd840609a45ec23e29865c86c6d15
Oct 30 23:05:43.598957 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:43.598941    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b805c0_763e_4cd9_b6f5_af98767b298e.slice/crio-acfa267edd58ba55c55464758813d6ab2e637fcc8e585829468c5668de2f3c8c WatchSource:0}: Error finding container acfa267edd58ba55c55464758813d6ab2e637fcc8e585829468c5668de2f3c8c: Status 404 returned error can't find the container with id acfa267edd58ba55c55464758813d6ab2e637fcc8e585829468c5668de2f3c8c
Oct 30 23:05:43.602712 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.602696    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:05:43.625440 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.625417    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:05:43.635434 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:43.635411    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22aabe7f_74b4_4cc7_b88b_1ec78aecc925.slice/crio-577bbe5f783c849e7733d86c2d6569bb0b9955764cc058b2a7c01a5fe3dda90c WatchSource:0}: Error finding container 577bbe5f783c849e7733d86c2d6569bb0b9955764cc058b2a7c01a5fe3dda90c: Status 404 returned error can't find the container with id 577bbe5f783c849e7733d86c2d6569bb0b9955764cc058b2a7c01a5fe3dda90c
Oct 30 23:05:43.646975 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.646957    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wk66h"
Oct 30 23:05:43.650258 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.650242    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:05:43.656169 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:43.656143    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91deced5_f22e_4148_98da_6a232be165c3.slice/crio-a4e5e6789fe136c0f61393a6b0b46b93a51333e25d04454ec1d97192e25b68d4 WatchSource:0}: Error finding container a4e5e6789fe136c0f61393a6b0b46b93a51333e25d04454ec1d97192e25b68d4: Status 404 returned error can't find the container with id a4e5e6789fe136c0f61393a6b0b46b93a51333e25d04454ec1d97192e25b68d4
Oct 30 23:05:43.658747 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:43.658660    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d5aef87_7d40_4aa2_958a_1fb73d427e3b.slice/crio-6619f18877b5bbac0bcd2088d544fb1fcb3733f5c8c3432c8f1bdec4bb752e3a WatchSource:0}: Error finding container 6619f18877b5bbac0bcd2088d544fb1fcb3733f5c8c3432c8f1bdec4bb752e3a: Status 404 returned error can't find the container with id 6619f18877b5bbac0bcd2088d544fb1fcb3733f5c8c3432c8f1bdec4bb752e3a
Oct 30 23:05:43.719710 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.719637    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:05:43.840472 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.840442    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs\") pod \"network-metrics-daemon-pln6h\" (UID: \"58ec1adc-599b-4796-b12e-1cc85a834b15\") " pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:43.840687 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.840489    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:43.840687 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.840596    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:05:43.840687 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.840628    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:05:43.840687 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.840668    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret podName:2e942185-68e3-4d57-9f42-49d2965b41e4 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:44.840654484 +0000 UTC m=+3.292273232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret") pod "global-pull-secret-syncer-cthfm" (UID: "2e942185-68e3-4d57-9f42-49d2965b41e4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:05:43.840856 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.840694    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs podName:58ec1adc-599b-4796-b12e-1cc85a834b15 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:44.840675995 +0000 UTC m=+3.292294741 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs") pod "network-metrics-daemon-pln6h" (UID: "58ec1adc-599b-4796-b12e-1cc85a834b15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:05:43.941858 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:43.941828    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46ww\" (UniqueName: \"kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww\") pod \"network-check-target-qvbgk\" (UID: \"70a51653-2167-4e11-bf04-81f5b10c9c9d\") " pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:05:43.942502 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.941972    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:05:43.942502 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.941987    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:05:43.942502 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.941998    2573 projected.go:196] Error preparing data for projected volume kube-api-access-q46ww for pod openshift-network-diagnostics/network-check-target-qvbgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:05:43.942502 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:43.942044    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww podName:70a51653-2167-4e11-bf04-81f5b10c9c9d nodeName:}" failed. No retries permitted until 2026-10-30 23:05:44.94203127 +0000 UTC m=+3.393650017 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-q46ww" (UniqueName: "kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww") pod "network-check-target-qvbgk" (UID: "70a51653-2167-4e11-bf04-81f5b10c9c9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:05:44.001374 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:44.001275    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79" event={"ID":"1918245f-b465-485f-9603-04ebe4846085","Type":"ContainerStarted","Data":"e8d2abb44c70942aa77687775e2477d94c0dd840609a45ec23e29865c86c6d15"}
Oct 30 23:05:44.002288 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:44.002261    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal" event={"ID":"a1b66be07f6b0e09e908458ab3790deb","Type":"ContainerStarted","Data":"9f28f4ffabe36c7eb7853ab824409f4c93fc666101154ea2a9de302865ebb19a"}
Oct 30 23:05:44.003354 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:44.003334    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2g2gr" event={"ID":"df39047e-a25e-4661-a34e-a6bca0f344f2","Type":"ContainerStarted","Data":"8d753418031592ca311fa0bc6cb8bcc039543c68e5ec696435c1dcbfc260c320"}
Oct 30 23:05:44.004456 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:44.004436    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z7d8k" event={"ID":"6d5aef87-7d40-4aa2-958a-1fb73d427e3b","Type":"ContainerStarted","Data":"6619f18877b5bbac0bcd2088d544fb1fcb3733f5c8c3432c8f1bdec4bb752e3a"}
Oct 30 23:05:44.005543 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:44.005526    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wk66h" event={"ID":"91deced5-f22e-4148-98da-6a232be165c3","Type":"ContainerStarted","Data":"a4e5e6789fe136c0f61393a6b0b46b93a51333e25d04454ec1d97192e25b68d4"}
Oct 30 23:05:44.006801 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:44.006774    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm" event={"ID":"22aabe7f-74b4-4cc7-b88b-1ec78aecc925","Type":"ContainerStarted","Data":"577bbe5f783c849e7733d86c2d6569bb0b9955764cc058b2a7c01a5fe3dda90c"}
Oct 30 23:05:44.007851 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:44.007824    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rcgrg" event={"ID":"81b805c0-763e-4cd9-b6f5-af98767b298e","Type":"ContainerStarted","Data":"acfa267edd58ba55c55464758813d6ab2e637fcc8e585829468c5668de2f3c8c"}
Oct 30 23:05:44.008919 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:44.008901    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal" event={"ID":"8bc6014254655639fce61607de5fe088","Type":"ContainerStarted","Data":"0c0ebf017ce4c7526dddc3e8f9d6f9518b84b7ca69a5085f1e40912bbfd5fc03"}
Oct 30 23:05:44.329634 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:44.329511    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:00:43 +0000 UTC" deadline="2028-07-16 06:49:55.33991853 +0000 UTC"
Oct 30 23:05:44.329634 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:44.329557    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14983h44m11.010364119s"
Oct 30 23:05:44.401246 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:44.401212    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108c7917_29b6_47bc_a590_35fd94c3de19.slice/crio-7de3c861ceb745ee60fac5d3355fbaf7a32b92c2101ba5aa447c370dbc1efe12 WatchSource:0}: Error finding container 7de3c861ceb745ee60fac5d3355fbaf7a32b92c2101ba5aa447c370dbc1efe12: Status 404 returned error can't find the container with id 7de3c861ceb745ee60fac5d3355fbaf7a32b92c2101ba5aa447c370dbc1efe12
Oct 30 23:05:44.418420 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:44.418396    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e14c04d_b583_4f0a_840c_d493b2a2bbaf.slice/crio-f88938488131bbe50049327147b95a87b12c1e36a3b0941d9062a772190f1524 WatchSource:0}: Error finding container f88938488131bbe50049327147b95a87b12c1e36a3b0941d9062a772190f1524: Status 404 returned error can't find the container with id f88938488131bbe50049327147b95a87b12c1e36a3b0941d9062a772190f1524
Oct 30 23:05:44.527724 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:05:44.527690    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29261ce8_6b07_4ea5_afa4_8500cc1c7bbf.slice/crio-7711a49b12888c6f8c173ce50e08a979cc1d841d1d0218af21cc633134a29005 WatchSource:0}: Error finding container 7711a49b12888c6f8c173ce50e08a979cc1d841d1d0218af21cc633134a29005: Status 404 returned error can't find the container with id 7711a49b12888c6f8c173ce50e08a979cc1d841d1d0218af21cc633134a29005
Oct 30 23:05:44.848960 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:44.848919    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs\") pod \"network-metrics-daemon-pln6h\" (UID: \"58ec1adc-599b-4796-b12e-1cc85a834b15\") " pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:44.849159 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:44.849010    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:44.849256 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:44.849179    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:05:44.849256 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:44.849249    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret podName:2e942185-68e3-4d57-9f42-49d2965b41e4 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:46.849226034 +0000 UTC m=+5.300844787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret") pod "global-pull-secret-syncer-cthfm" (UID: "2e942185-68e3-4d57-9f42-49d2965b41e4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:05:44.849427 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:44.849319    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:05:44.849427 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:44.849354    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs podName:58ec1adc-599b-4796-b12e-1cc85a834b15 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:46.849341601 +0000 UTC m=+5.300960350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs") pod "network-metrics-daemon-pln6h" (UID: "58ec1adc-599b-4796-b12e-1cc85a834b15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:05:44.950124 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:44.950080    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46ww\" (UniqueName: \"kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww\") pod \"network-check-target-qvbgk\" (UID: \"70a51653-2167-4e11-bf04-81f5b10c9c9d\") " pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:05:44.950624 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:44.950260    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:05:44.950624 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:44.950280    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:05:44.950624 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:44.950295    2573 projected.go:196] Error preparing data for projected volume kube-api-access-q46ww for pod openshift-network-diagnostics/network-check-target-qvbgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:05:44.950624 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:44.950359    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww podName:70a51653-2167-4e11-bf04-81f5b10c9c9d nodeName:}" failed. No retries permitted until 2026-10-30 23:05:46.950337698 +0000 UTC m=+5.401956469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-q46ww" (UniqueName: "kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww") pod "network-check-target-qvbgk" (UID: "70a51653-2167-4e11-bf04-81f5b10c9c9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:05:44.996412 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:44.995909    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:44.996412 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:44.996039    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cthfm" podUID="2e942185-68e3-4d57-9f42-49d2965b41e4"
Oct 30 23:05:44.996412 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:44.996125    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:44.996412 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:44.996218    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pln6h" podUID="58ec1adc-599b-4796-b12e-1cc85a834b15"
Oct 30 23:05:44.996412 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:44.996279    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:05:44.996412 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:44.996349    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvbgk" podUID="70a51653-2167-4e11-bf04-81f5b10c9c9d"
Oct 30 23:05:45.015216 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:45.015148    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fg8fg" event={"ID":"7e14c04d-b583-4f0a-840c-d493b2a2bbaf","Type":"ContainerStarted","Data":"f88938488131bbe50049327147b95a87b12c1e36a3b0941d9062a772190f1524"}
Oct 30 23:05:45.023186 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:45.023134    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gldcx" event={"ID":"108c7917-29b6-47bc-a590-35fd94c3de19","Type":"ContainerStarted","Data":"7de3c861ceb745ee60fac5d3355fbaf7a32b92c2101ba5aa447c370dbc1efe12"}
Oct 30 23:05:45.029413 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:45.029380    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dzklv" event={"ID":"29261ce8-6b07-4ea5-afa4-8500cc1c7bbf","Type":"ContainerStarted","Data":"7711a49b12888c6f8c173ce50e08a979cc1d841d1d0218af21cc633134a29005"}
Oct 30 23:05:45.330787 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:45.330690    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:00:43 +0000 UTC" deadline="2028-05-19 07:06:40.524351449 +0000 UTC"
Oct 30 23:05:45.330787 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:45.330730    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13592h0m55.193626384s"
Oct 30 23:05:46.868641 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:46.868585    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:46.868997 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:46.868718    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs\") pod \"network-metrics-daemon-pln6h\" (UID: \"58ec1adc-599b-4796-b12e-1cc85a834b15\") " pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:46.868997 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:46.868886    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:05:46.868997 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:46.868951    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs podName:58ec1adc-599b-4796-b12e-1cc85a834b15 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:50.868932768 +0000 UTC m=+9.320551518 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs") pod "network-metrics-daemon-pln6h" (UID: "58ec1adc-599b-4796-b12e-1cc85a834b15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:05:46.869114 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:46.869023    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:05:46.869114 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:46.869058    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret podName:2e942185-68e3-4d57-9f42-49d2965b41e4 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:50.869046047 +0000 UTC m=+9.320664799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret") pod "global-pull-secret-syncer-cthfm" (UID: "2e942185-68e3-4d57-9f42-49d2965b41e4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:05:46.969619 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:46.969579    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46ww\" (UniqueName: \"kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww\") pod \"network-check-target-qvbgk\" (UID: \"70a51653-2167-4e11-bf04-81f5b10c9c9d\") " pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:05:46.969877 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:46.969858    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:05:46.969930 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:46.969886    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:05:46.969930 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:46.969901    2573 projected.go:196] Error preparing data for projected volume kube-api-access-q46ww for pod openshift-network-diagnostics/network-check-target-qvbgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:05:46.970019 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:46.969966    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww podName:70a51653-2167-4e11-bf04-81f5b10c9c9d nodeName:}" failed. No retries permitted until 2026-10-30 23:05:50.969945087 +0000 UTC m=+9.421563859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-q46ww" (UniqueName: "kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww") pod "network-check-target-qvbgk" (UID: "70a51653-2167-4e11-bf04-81f5b10c9c9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:05:46.996710 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:46.996676    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:46.996896 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:46.996799    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pln6h" podUID="58ec1adc-599b-4796-b12e-1cc85a834b15"
Oct 30 23:05:46.996971 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:46.996961    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:05:46.997057 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:46.997032    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvbgk" podUID="70a51653-2167-4e11-bf04-81f5b10c9c9d"
Oct 30 23:05:46.997128 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:46.997072    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:46.997128 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:46.997116    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cthfm" podUID="2e942185-68e3-4d57-9f42-49d2965b41e4"
Oct 30 23:05:48.076285 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:48.076029    2573 generic.go:356] "Generic (PLEG): container finished" podID="6d5aef87-7d40-4aa2-958a-1fb73d427e3b" containerID="907cbd329009711b667f1ef4fd482bf3893f4af473e99633952dfaea0d7cbe5c" exitCode=0
Oct 30 23:05:48.076821 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:48.076312    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z7d8k" event={"ID":"6d5aef87-7d40-4aa2-958a-1fb73d427e3b","Type":"ContainerDied","Data":"907cbd329009711b667f1ef4fd482bf3893f4af473e99633952dfaea0d7cbe5c"}
Oct 30 23:05:48.996876 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:48.996769    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:48.997048 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:48.996920    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pln6h" podUID="58ec1adc-599b-4796-b12e-1cc85a834b15"
Oct 30 23:05:48.997333 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:48.997312    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:05:48.997447 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:48.997415    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvbgk" podUID="70a51653-2167-4e11-bf04-81f5b10c9c9d"
Oct 30 23:05:48.997509 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:48.997465    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:48.997568 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:48.997538    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cthfm" podUID="2e942185-68e3-4d57-9f42-49d2965b41e4"
Oct 30 23:05:49.088551 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:49.088502    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal" event={"ID":"a1b66be07f6b0e09e908458ab3790deb","Type":"ContainerStarted","Data":"9e5235d72e5e721ca45d2fff3566007931ae9bc547833d1498aa2c2ebb7f8c18"}
Oct 30 23:05:49.097467 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:49.097428    2573 generic.go:356] "Generic (PLEG): container finished" podID="8bc6014254655639fce61607de5fe088" containerID="2c05da7699bbb3c4abc65436e7b2d424b7e38acd1d39300ddc3bc6f0c3029a73" exitCode=0
Oct 30 23:05:49.097612 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:49.097488    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal" event={"ID":"8bc6014254655639fce61607de5fe088","Type":"ContainerDied","Data":"2c05da7699bbb3c4abc65436e7b2d424b7e38acd1d39300ddc3bc6f0c3029a73"}
Oct 30 23:05:49.356855 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:49.353024    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal" podStartSLOduration=6.352993609 podStartE2EDuration="6.352993609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:49.216273592 +0000 UTC m=+7.667892360" watchObservedRunningTime="2026-10-30 23:05:49.352993609 +0000 UTC m=+7.804612382"
Oct 30 23:05:50.103698 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:50.103532    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal" event={"ID":"8bc6014254655639fce61607de5fe088","Type":"ContainerStarted","Data":"83ca72038a6ecbeb4c2cf5d1e0d543ef3744f4bce8e946dba4d340a2f583ff38"}
Oct 30 23:05:50.106756 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:50.106726    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-fg8fg" event={"ID":"7e14c04d-b583-4f0a-840c-d493b2a2bbaf","Type":"ContainerStarted","Data":"39441e03e44e7cf5ee4c1b96a51efb358039427e448c2dc4c5f45a2b2ebe84ee"}
Oct 30 23:05:50.356461 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:50.356148    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal" podStartSLOduration=7.356126132 podStartE2EDuration="7.356126132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:50.229888958 +0000 UTC m=+8.681507729" watchObservedRunningTime="2026-10-30 23:05:50.356126132 +0000 UTC m=+8.807744905"
Oct 30 23:05:50.906238 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:50.906197    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:50.906374 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:50.906296    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs\") pod \"network-metrics-daemon-pln6h\" (UID: \"58ec1adc-599b-4796-b12e-1cc85a834b15\") " pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:50.906374 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:50.906344    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:05:50.906493 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:50.906408    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret podName:2e942185-68e3-4d57-9f42-49d2965b41e4 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:58.906388533 +0000 UTC m=+17.358007300 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret") pod "global-pull-secret-syncer-cthfm" (UID: "2e942185-68e3-4d57-9f42-49d2965b41e4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:05:50.906493 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:50.906415    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:05:50.906493 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:50.906473    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs podName:58ec1adc-599b-4796-b12e-1cc85a834b15 nodeName:}" failed. No retries permitted until 2026-10-30 23:05:58.906457246 +0000 UTC m=+17.358075996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs") pod "network-metrics-daemon-pln6h" (UID: "58ec1adc-599b-4796-b12e-1cc85a834b15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:05:50.996638 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:50.996121    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:50.996638 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:50.996274    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pln6h" podUID="58ec1adc-599b-4796-b12e-1cc85a834b15"
Oct 30 23:05:50.996638 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:50.996353    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:05:50.996638 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:50.996431    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvbgk" podUID="70a51653-2167-4e11-bf04-81f5b10c9c9d"
Oct 30 23:05:50.996638 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:50.996488    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:50.996638 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:50.996564    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cthfm" podUID="2e942185-68e3-4d57-9f42-49d2965b41e4"
Oct 30 23:05:51.007916 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:51.007579    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46ww\" (UniqueName: \"kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww\") pod \"network-check-target-qvbgk\" (UID: \"70a51653-2167-4e11-bf04-81f5b10c9c9d\") " pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:05:51.007916 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:51.007780    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:05:51.007916 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:51.007801    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:05:51.007916 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:51.007815    2573 projected.go:196] Error preparing data for projected volume kube-api-access-q46ww for pod openshift-network-diagnostics/network-check-target-qvbgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:05:51.007916 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:51.007872    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww podName:70a51653-2167-4e11-bf04-81f5b10c9c9d nodeName:}" failed. No retries permitted until 2026-10-30 23:05:59.007851979 +0000 UTC m=+17.459470750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-q46ww" (UniqueName: "kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww") pod "network-check-target-qvbgk" (UID: "70a51653-2167-4e11-bf04-81f5b10c9c9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:05:51.112869 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:51.112799    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2g2gr" event={"ID":"df39047e-a25e-4661-a34e-a6bca0f344f2","Type":"ContainerStarted","Data":"c60322da62b628446949bc28a23c7b2123077f2d75f3d4a496e1945bf0260b17"}
Oct 30 23:05:51.116686 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:51.116277    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79" event={"ID":"1918245f-b465-485f-9603-04ebe4846085","Type":"ContainerStarted","Data":"582df1f6ef5fe584faad7b11a2c9745d651c2cbc5f3686c57abb2213e7de2c20"}
Oct 30 23:05:51.238542 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:51.238436    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-fg8fg" podStartSLOduration=2.953127494 podStartE2EDuration="8.238411093s" totalImagesPullingTime="5.285283599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:50.3556392 +0000 UTC m=+8.807257973" watchObservedRunningTime="2026-10-30 23:05:51.238411093 +0000 UTC m=+9.690029865"
Oct 30 23:05:51.268155 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:51.268113    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-fg8fg"
Oct 30 23:05:51.268700 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:51.268673    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-fg8fg"
Oct 30 23:05:51.394614 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:51.394548    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2g2gr" podStartSLOduration=1.8708371179999999 podStartE2EDuration="8.394533495s" totalImagesPullingTime="6.523696377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:51.239597432 +0000 UTC m=+9.691216204" watchObservedRunningTime="2026-10-30 23:05:51.394533495 +0000 UTC m=+9.846152265"
Oct 30 23:05:52.119177 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:52.118925    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-fg8fg"
Oct 30 23:05:52.119703 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:52.119566    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fg8fg"
Oct 30 23:05:52.997476 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:52.996803    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:52.997476 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:52.996820    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:05:52.997476 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:52.996803    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:52.997476 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:52.996921    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pln6h" podUID="58ec1adc-599b-4796-b12e-1cc85a834b15"
Oct 30 23:05:52.997476 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:52.996963    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cthfm" podUID="2e942185-68e3-4d57-9f42-49d2965b41e4"
Oct 30 23:05:52.997476 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:52.996971    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvbgk" podUID="70a51653-2167-4e11-bf04-81f5b10c9c9d"
Oct 30 23:05:53.016012 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:53.015981    2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 23:05:53.131898 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:53.131825    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79" event={"ID":"1918245f-b465-485f-9603-04ebe4846085","Type":"ContainerStarted","Data":"33120588d2363d7a4d91929a9e8ebbc5146750a0706ba841280d0b0851932a95"}
Oct 30 23:05:54.011340 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:54.010843    2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:05:53.016009058Z","UUID":"224b2c59-9781-4a96-a3da-69790da6dffc","Handler":null,"Name":"","Endpoint":""}
Oct 30 23:05:54.012633 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:54.012590    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 23:05:54.012800 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:54.012643    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 23:05:54.139254 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:54.139208    2573 generic.go:356] "Generic (PLEG): container finished" podID="6d5aef87-7d40-4aa2-958a-1fb73d427e3b" containerID="31dd7921dfbc997e0fd40c328c1095805daea044b6d5758886238c28d6d9b797" exitCode=0
Oct 30 23:05:54.139771 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:54.139312    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z7d8k" event={"ID":"6d5aef87-7d40-4aa2-958a-1fb73d427e3b","Type":"ContainerDied","Data":"31dd7921dfbc997e0fd40c328c1095805daea044b6d5758886238c28d6d9b797"}
Oct 30 23:05:54.142439 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:54.142412    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gldcx" event={"ID":"108c7917-29b6-47bc-a590-35fd94c3de19","Type":"ContainerStarted","Data":"6e481055b71ce9589c5ac0cb2702ea0b56f00b8ea8f9b31b0ad7798f7d267d3a"}
Oct 30 23:05:54.146545 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:54.146520    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wk66h" event={"ID":"91deced5-f22e-4148-98da-6a232be165c3","Type":"ContainerStarted","Data":"11155ac2bac539cc57cd979adf8a087ef5d927cfcf2faea623ce97efee88d2cc"}
Oct 30 23:05:54.387676 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:54.387556    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gldcx" podStartSLOduration=2.40746262 podStartE2EDuration="11.387539522s" totalImagesPullingTime="8.980076902s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:54.386688387 +0000 UTC m=+12.838307155" watchObservedRunningTime="2026-10-30 23:05:54.387539522 +0000 UTC m=+12.839158290"
Oct 30 23:05:54.528029 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:54.527976    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-wk66h" podStartSLOduration=2.023942231 podStartE2EDuration="11.527962437s" totalImagesPullingTime="9.504020206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:54.52733469 +0000 UTC m=+12.978953461" watchObservedRunningTime="2026-10-30 23:05:54.527962437 +0000 UTC m=+12.979581203"
Oct 30 23:05:54.995750 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:54.995723    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:54.995839 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:54.995752    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:05:54.995889 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:54.995858    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pln6h" podUID="58ec1adc-599b-4796-b12e-1cc85a834b15"
Oct 30 23:05:54.995959 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:54.995940    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvbgk" podUID="70a51653-2167-4e11-bf04-81f5b10c9c9d"
Oct 30 23:05:54.996019 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:54.995995    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:54.996084 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:54.996066    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cthfm" podUID="2e942185-68e3-4d57-9f42-49d2965b41e4"
Oct 30 23:05:55.149279 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:55.149240    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dzklv" event={"ID":"29261ce8-6b07-4ea5-afa4-8500cc1c7bbf","Type":"ContainerStarted","Data":"ed4388fb137f764fc16f232b18d67678239b0e81905aa8bcdaa61dc9a5230d5f"}
Oct 30 23:05:55.264205 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:55.264123    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dzklv" podStartSLOduration=3.308707748 podStartE2EDuration="12.264106384s" totalImagesPullingTime="8.955398636s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:55.263534882 +0000 UTC m=+13.715153662" watchObservedRunningTime="2026-10-30 23:05:55.264106384 +0000 UTC m=+13.715725155"
Oct 30 23:05:56.153380 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:56.153294    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79" event={"ID":"1918245f-b465-485f-9603-04ebe4846085","Type":"ContainerStarted","Data":"1bd24cbba33df69e43503a19b307015551afc74ebbe5576f5c92964ca9e94571"}
Oct 30 23:05:56.155699 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:56.155673    2573 generic.go:356] "Generic (PLEG): container finished" podID="6d5aef87-7d40-4aa2-958a-1fb73d427e3b" containerID="2b944907ecda24ca4f78216a09a75e77902a3631bf71f84789f0ed571e22e3c7" exitCode=0
Oct 30 23:05:56.155822 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:56.155749    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z7d8k" event={"ID":"6d5aef87-7d40-4aa2-958a-1fb73d427e3b","Type":"ContainerDied","Data":"2b944907ecda24ca4f78216a09a75e77902a3631bf71f84789f0ed571e22e3c7"}
Oct 30 23:05:56.157856 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:56.157835    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-rcgrg" event={"ID":"81b805c0-763e-4cd9-b6f5-af98767b298e","Type":"ContainerStarted","Data":"ab31fe150de57e9eba03386b5443ed70a8fd8b392eb4d724da7fcbc1677e1426"}
Oct 30 23:05:56.271572 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:56.271513    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79" podStartSLOduration=2.177919554 podStartE2EDuration="13.271498677s" totalImagesPullingTime="11.093579123s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:56.271457109 +0000 UTC m=+14.723075888" watchObservedRunningTime="2026-10-30 23:05:56.271498677 +0000 UTC m=+14.723117443"
Oct 30 23:05:56.385998 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:56.385945    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-rcgrg" podStartSLOduration=1.504960424 podStartE2EDuration="13.385929792s" totalImagesPullingTime="11.880969368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:05:56.385902243 +0000 UTC m=+14.837521020" watchObservedRunningTime="2026-10-30 23:05:56.385929792 +0000 UTC m=+14.837548553"
Oct 30 23:05:56.995725 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:56.995695    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:56.995900 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:56.995806    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pln6h" podUID="58ec1adc-599b-4796-b12e-1cc85a834b15"
Oct 30 23:05:56.996186 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:56.996161    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:05:56.996293 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:56.996259    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvbgk" podUID="70a51653-2167-4e11-bf04-81f5b10c9c9d"
Oct 30 23:05:56.996334 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:56.996315    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:56.996370 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:56.996361    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cthfm" podUID="2e942185-68e3-4d57-9f42-49d2965b41e4"
Oct 30 23:05:58.165370 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:58.165335    2573 generic.go:356] "Generic (PLEG): container finished" podID="6d5aef87-7d40-4aa2-958a-1fb73d427e3b" containerID="46d7dd8e9987eb646077f0f9458e152722cfd4f4331d250672e87bf084c6c5dc" exitCode=0
Oct 30 23:05:58.165952 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:58.165392    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z7d8k" event={"ID":"6d5aef87-7d40-4aa2-958a-1fb73d427e3b","Type":"ContainerDied","Data":"46d7dd8e9987eb646077f0f9458e152722cfd4f4331d250672e87bf084c6c5dc"}
Oct 30 23:05:58.969406 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:58.969185    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:58.969636 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:58.969461    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs\") pod \"network-metrics-daemon-pln6h\" (UID: \"58ec1adc-599b-4796-b12e-1cc85a834b15\") " pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:58.969636 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:58.969338    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:05:58.969636 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:58.969542    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret podName:2e942185-68e3-4d57-9f42-49d2965b41e4 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:14.969523844 +0000 UTC m=+33.421142609 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret") pod "global-pull-secret-syncer-cthfm" (UID: "2e942185-68e3-4d57-9f42-49d2965b41e4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:05:58.969636 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:58.969561    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:05:58.969636 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:58.969614    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs podName:58ec1adc-599b-4796-b12e-1cc85a834b15 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:14.969583906 +0000 UTC m=+33.421202651 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs") pod "network-metrics-daemon-pln6h" (UID: "58ec1adc-599b-4796-b12e-1cc85a834b15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:05:59.003323 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:59.003284    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:05:59.003323 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:59.003313    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:05:59.003537 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:59.003313    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:05:59.003537 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:59.003436    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pln6h" podUID="58ec1adc-599b-4796-b12e-1cc85a834b15"
Oct 30 23:05:59.003656 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:59.003634    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cthfm" podUID="2e942185-68e3-4d57-9f42-49d2965b41e4"
Oct 30 23:05:59.003744 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:59.003721    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvbgk" podUID="70a51653-2167-4e11-bf04-81f5b10c9c9d"
Oct 30 23:05:59.070706 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:05:59.070670    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46ww\" (UniqueName: \"kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww\") pod \"network-check-target-qvbgk\" (UID: \"70a51653-2167-4e11-bf04-81f5b10c9c9d\") " pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:05:59.070883 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:59.070866    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 23:05:59.070950 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:59.070893    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 23:05:59.070950 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:59.070906    2573 projected.go:196] Error preparing data for projected volume kube-api-access-q46ww for pod openshift-network-diagnostics/network-check-target-qvbgk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:05:59.071036 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:05:59.070973    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww podName:70a51653-2167-4e11-bf04-81f5b10c9c9d nodeName:}" failed. No retries permitted until 2026-10-30 23:06:15.070955125 +0000 UTC m=+33.522573884 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-q46ww" (UniqueName: "kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww") pod "network-check-target-qvbgk" (UID: "70a51653-2167-4e11-bf04-81f5b10c9c9d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 23:06:00.172197 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:00.172163    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm" event={"ID":"22aabe7f-74b4-4cc7-b88b-1ec78aecc925","Type":"ContainerStarted","Data":"78f315cd1b21d5c95273143b891ae602ff313486b014fc4463a296d172c3e65f"}
Oct 30 23:06:00.172508 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:00.172288    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm" event={"ID":"22aabe7f-74b4-4cc7-b88b-1ec78aecc925","Type":"ContainerStarted","Data":"80cc88d7c5f765ba1e0bef5b425d52b0be26125f185d46f22e5c733aee7c542d"}
Oct 30 23:06:00.172508 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:00.172338    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm" event={"ID":"22aabe7f-74b4-4cc7-b88b-1ec78aecc925","Type":"ContainerStarted","Data":"ec4e3ebac9168f7a1193a9e18cd4f9b8d1d2ffd023d11e6c1b2a0160404d28a4"}
Oct 30 23:06:00.172508 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:00.172383    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm" event={"ID":"22aabe7f-74b4-4cc7-b88b-1ec78aecc925","Type":"ContainerStarted","Data":"f7e74b2c200fdfcba77139d9b3a07befa7f2cb6af8875cf97c235b6c869de182"}
Oct 30 23:06:00.995983 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:00.995814    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:06:00.996112 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:00.995841    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:00.996112 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:00.996089    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pln6h" podUID="58ec1adc-599b-4796-b12e-1cc85a834b15"
Oct 30 23:06:00.996236 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:00.995847    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:06:00.996236 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:00.996125    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cthfm" podUID="2e942185-68e3-4d57-9f42-49d2965b41e4"
Oct 30 23:06:00.996236 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:00.996220    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvbgk" podUID="70a51653-2167-4e11-bf04-81f5b10c9c9d"
Oct 30 23:06:01.176831 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:01.176792    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm" event={"ID":"22aabe7f-74b4-4cc7-b88b-1ec78aecc925","Type":"ContainerStarted","Data":"c3cd197cc378fe86871bf6e5efaaca3aaaa52887e343f2502eff5015e2bdd81b"}
Oct 30 23:06:01.177228 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:01.176868    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm" event={"ID":"22aabe7f-74b4-4cc7-b88b-1ec78aecc925","Type":"ContainerStarted","Data":"8943ae02593611632e71497644b4a76188b6ee08fcb6e2ee61935afb3326d5f5"}
Oct 30 23:06:01.179333 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:01.179308    2573 generic.go:356] "Generic (PLEG): container finished" podID="6d5aef87-7d40-4aa2-958a-1fb73d427e3b" containerID="4aff3584c5c385b309542300b7adecf952f644dd455110ecfb2d954875e27805" exitCode=0
Oct 30 23:06:01.179429 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:01.179344    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z7d8k" event={"ID":"6d5aef87-7d40-4aa2-958a-1fb73d427e3b","Type":"ContainerDied","Data":"4aff3584c5c385b309542300b7adecf952f644dd455110ecfb2d954875e27805"}
Oct 30 23:06:02.184094 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:02.184059    2573 generic.go:356] "Generic (PLEG): container finished" podID="6d5aef87-7d40-4aa2-958a-1fb73d427e3b" containerID="83701692091b14f4ea64ac22ecf562beede8e17c5efc134cc2b4081329d48060" exitCode=0
Oct 30 23:06:02.184540 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:02.184117    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z7d8k" event={"ID":"6d5aef87-7d40-4aa2-958a-1fb73d427e3b","Type":"ContainerDied","Data":"83701692091b14f4ea64ac22ecf562beede8e17c5efc134cc2b4081329d48060"}
Oct 30 23:06:02.996614 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:02.996568    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:06:02.996766 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:02.996657    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:06:02.996766 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:02.996681    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:02.996766 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:02.996753    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pln6h" podUID="58ec1adc-599b-4796-b12e-1cc85a834b15"
Oct 30 23:06:02.996934 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:02.996796    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cthfm" podUID="2e942185-68e3-4d57-9f42-49d2965b41e4"
Oct 30 23:06:02.996934 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:02.996855    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvbgk" podUID="70a51653-2167-4e11-bf04-81f5b10c9c9d"
Oct 30 23:06:03.188526 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:03.188489    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm" event={"ID":"22aabe7f-74b4-4cc7-b88b-1ec78aecc925","Type":"ContainerStarted","Data":"89e6ebc355ddfbe8dbb856036e92bde5d825e99b3251427db8df825ba1324b9f"}
Oct 30 23:06:03.191768 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:03.191744    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-z7d8k" event={"ID":"6d5aef87-7d40-4aa2-958a-1fb73d427e3b","Type":"ContainerStarted","Data":"e5ab1cc41f8b9c7ae1c0c1e26f2e6b77605be0171ab2f938792665d4eea7250b"}
Oct 30 23:06:03.313696 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:03.313615    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-z7d8k" podStartSLOduration=6.857182143 podStartE2EDuration="20.31358498s" totalImagesPullingTime="13.456402837s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:03.31352815 +0000 UTC m=+21.765146918" watchObservedRunningTime="2026-10-30 23:06:03.31358498 +0000 UTC m=+21.765203747"
Oct 30 23:06:04.996117 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:04.996080    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:04.996977 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:04.996119    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:06:04.996977 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:04.996086    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:06:04.996977 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:04.996179    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cthfm" podUID="2e942185-68e3-4d57-9f42-49d2965b41e4"
Oct 30 23:06:04.996977 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:04.996283    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pln6h" podUID="58ec1adc-599b-4796-b12e-1cc85a834b15"
Oct 30 23:06:04.996977 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:04.996352    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvbgk" podUID="70a51653-2167-4e11-bf04-81f5b10c9c9d"
Oct 30 23:06:05.202361 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:05.202325    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm" event={"ID":"22aabe7f-74b4-4cc7-b88b-1ec78aecc925","Type":"ContainerStarted","Data":"a7835cfd1f6d5f3e9647621d60981d8a1d540e51712d98624481b564e4f42a63"}
Oct 30 23:06:05.202663 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:05.202630    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:05.218551 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:05.218528    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:05.481499 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:05.481405    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm" podStartSLOduration=6.193230628 podStartE2EDuration="22.481393965s" totalImagesPullingTime="16.288163337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:05.481155265 +0000 UTC m=+23.932774045" watchObservedRunningTime="2026-10-30 23:06:05.481393965 +0000 UTC m=+23.933012732"
Oct 30 23:06:06.205010 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:06.204986    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:06:06.205417 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:06.205402    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:06.221009 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:06.220987    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:06.995991 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:06.995730    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:06:06.996175 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:06.995783    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:06:06.996175 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:06.996035    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pln6h" podUID="58ec1adc-599b-4796-b12e-1cc85a834b15"
Oct 30 23:06:06.996175 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:06.995790    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:06.996175 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:06.996094    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvbgk" podUID="70a51653-2167-4e11-bf04-81f5b10c9c9d"
Oct 30 23:06:06.996382 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:06.996190    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cthfm" podUID="2e942185-68e3-4d57-9f42-49d2965b41e4"
Oct 30 23:06:07.056973 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:07.056944    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cthfm"]
Oct 30 23:06:07.060137 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:07.060111    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qvbgk"]
Oct 30 23:06:07.060695 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:07.060679    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pln6h"]
Oct 30 23:06:07.206715 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:07.206689    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:06:07.207059 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:07.206775    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:06:07.207059 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:07.206689    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:07.207059 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:07.206803    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pln6h" podUID="58ec1adc-599b-4796-b12e-1cc85a834b15"
Oct 30 23:06:07.207059 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:07.206690    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:06:07.207238 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:07.207198    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cthfm" podUID="2e942185-68e3-4d57-9f42-49d2965b41e4"
Oct 30 23:06:07.207312 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:07.207292    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvbgk" podUID="70a51653-2167-4e11-bf04-81f5b10c9c9d"
Oct 30 23:06:08.208704 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:08.208677    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:06:08.891378 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:08.891353    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:08.996111 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:08.996079    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:06:08.996259 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:08.996126    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:08.996259 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:08.996224    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:06:08.996259 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:08.996229    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pln6h" podUID="58ec1adc-599b-4796-b12e-1cc85a834b15"
Oct 30 23:06:08.996365 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:08.996301    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvbgk" podUID="70a51653-2167-4e11-bf04-81f5b10c9c9d"
Oct 30 23:06:08.996365 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:08.996348    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cthfm" podUID="2e942185-68e3-4d57-9f42-49d2965b41e4"
Oct 30 23:06:09.223820 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:09.223740    2573 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm" podUID="22aabe7f-74b4-4cc7-b88b-1ec78aecc925" containerName="ovnkube-controller" probeResult="failure" output=""
Oct 30 23:06:10.996047 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:10.995963    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:06:10.996439 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:10.996164    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:06:10.996439 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:10.996185    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:10.996439 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:10.996285    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pln6h" podUID="58ec1adc-599b-4796-b12e-1cc85a834b15"
Oct 30 23:06:10.996439 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:10.996328    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvbgk" podUID="70a51653-2167-4e11-bf04-81f5b10c9c9d"
Oct 30 23:06:10.996579 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:10.996443    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cthfm" podUID="2e942185-68e3-4d57-9f42-49d2965b41e4"
Oct 30 23:06:12.996812 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:12.996628    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:06:12.997182 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:12.996629    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:12.997182 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:12.996915    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-pln6h" podUID="58ec1adc-599b-4796-b12e-1cc85a834b15"
Oct 30 23:06:12.997182 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:12.996953    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cthfm" podUID="2e942185-68e3-4d57-9f42-49d2965b41e4"
Oct 30 23:06:12.997182 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:12.996652    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:06:12.997182 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:12.997031    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qvbgk" podUID="70a51653-2167-4e11-bf04-81f5b10c9c9d"
Oct 30 23:06:13.376069 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.376041    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeReady"
Oct 30 23:06:13.376229 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.376146    2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 23:06:13.471671 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.471642    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xslb9"]
Oct 30 23:06:13.474622 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.474607    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xslb9"
Oct 30 23:06:13.478754 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.478731    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8qhk5"]
Oct 30 23:06:13.481878 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.481758    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:13.527611 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.527562    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 23:06:13.527776 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.527756    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-l6bwx\""
Oct 30 23:06:13.527816 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.527758    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 23:06:13.527896 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.527885    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 23:06:13.538723 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.538705    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 23:06:13.538843 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.538832    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 23:06:13.538923 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.538912    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cqcdm\""
Oct 30 23:06:13.551210 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.551188    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xslb9"]
Oct 30 23:06:13.575829 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.575805    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2fzc\" (UniqueName: \"kubernetes.io/projected/4e4bb115-b334-423e-86f1-1e1abb499b5d-kube-api-access-l2fzc\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:13.575829 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.575830    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b99dd331-1d6d-434c-9b02-1b7694955006-cert\") pod \"ingress-canary-xslb9\" (UID: \"b99dd331-1d6d-434c-9b02-1b7694955006\") " pod="openshift-ingress-canary/ingress-canary-xslb9"
Oct 30 23:06:13.575958 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.575846    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e4bb115-b334-423e-86f1-1e1abb499b5d-tmp-dir\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:13.575958 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.575876    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bm7q\" (UniqueName: \"kubernetes.io/projected/b99dd331-1d6d-434c-9b02-1b7694955006-kube-api-access-5bm7q\") pod \"ingress-canary-xslb9\" (UID: \"b99dd331-1d6d-434c-9b02-1b7694955006\") " pod="openshift-ingress-canary/ingress-canary-xslb9"
Oct 30 23:06:13.575958 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.575926    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:13.576046 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.575958    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4bb115-b334-423e-86f1-1e1abb499b5d-config-volume\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:13.612994 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.612970    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8qhk5"]
Oct 30 23:06:13.676308 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.676249    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4bb115-b334-423e-86f1-1e1abb499b5d-config-volume\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:13.676308 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.676302    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2fzc\" (UniqueName: \"kubernetes.io/projected/4e4bb115-b334-423e-86f1-1e1abb499b5d-kube-api-access-l2fzc\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:13.676470 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.676323    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b99dd331-1d6d-434c-9b02-1b7694955006-cert\") pod \"ingress-canary-xslb9\" (UID: \"b99dd331-1d6d-434c-9b02-1b7694955006\") " pod="openshift-ingress-canary/ingress-canary-xslb9"
Oct 30 23:06:13.676470 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.676443    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e4bb115-b334-423e-86f1-1e1abb499b5d-tmp-dir\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:13.676567 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.676514    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bm7q\" (UniqueName: \"kubernetes.io/projected/b99dd331-1d6d-434c-9b02-1b7694955006-kube-api-access-5bm7q\") pod \"ingress-canary-xslb9\" (UID: \"b99dd331-1d6d-434c-9b02-1b7694955006\") " pod="openshift-ingress-canary/ingress-canary-xslb9"
Oct 30 23:06:13.676567 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.676539    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:13.676693 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:13.676667    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:06:13.676745 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:13.676730    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls podName:4e4bb115-b334-423e-86f1-1e1abb499b5d nodeName:}" failed. No retries permitted until 2026-10-30 23:06:14.176712416 +0000 UTC m=+32.628331179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls") pod "dns-default-8qhk5" (UID: "4e4bb115-b334-423e-86f1-1e1abb499b5d") : secret "dns-default-metrics-tls" not found
Oct 30 23:06:13.676803 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.676776    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e4bb115-b334-423e-86f1-1e1abb499b5d-tmp-dir\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:13.676987 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.676965    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4bb115-b334-423e-86f1-1e1abb499b5d-config-volume\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:13.683225 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.680762    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b99dd331-1d6d-434c-9b02-1b7694955006-cert\") pod \"ingress-canary-xslb9\" (UID: \"b99dd331-1d6d-434c-9b02-1b7694955006\") " pod="openshift-ingress-canary/ingress-canary-xslb9"
Oct 30 23:06:13.735233 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.735208    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2fzc\" (UniqueName: \"kubernetes.io/projected/4e4bb115-b334-423e-86f1-1e1abb499b5d-kube-api-access-l2fzc\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:13.735722 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.735705    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bm7q\" (UniqueName: \"kubernetes.io/projected/b99dd331-1d6d-434c-9b02-1b7694955006-kube-api-access-5bm7q\") pod \"ingress-canary-xslb9\" (UID: \"b99dd331-1d6d-434c-9b02-1b7694955006\") " pod="openshift-ingress-canary/ingress-canary-xslb9"
Oct 30 23:06:13.786670 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:13.786650    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xslb9"
Oct 30 23:06:14.162210 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:14.162178    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xslb9"]
Oct 30 23:06:14.165230 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:06:14.165202    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb99dd331_1d6d_434c_9b02_1b7694955006.slice/crio-3aba3250bfb20a60bc07e992dab181daa77b3a0019b1db9e6ad7c59a05535b39 WatchSource:0}: Error finding container 3aba3250bfb20a60bc07e992dab181daa77b3a0019b1db9e6ad7c59a05535b39: Status 404 returned error can't find the container with id 3aba3250bfb20a60bc07e992dab181daa77b3a0019b1db9e6ad7c59a05535b39
Oct 30 23:06:14.181003 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:14.180984    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:14.181128 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:14.181113    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:06:14.181172 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:14.181166    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls podName:4e4bb115-b334-423e-86f1-1e1abb499b5d nodeName:}" failed. No retries permitted until 2026-10-30 23:06:15.181151439 +0000 UTC m=+33.632770200 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls") pod "dns-default-8qhk5" (UID: "4e4bb115-b334-423e-86f1-1e1abb499b5d") : secret "dns-default-metrics-tls" not found
Oct 30 23:06:14.222717 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:14.222685    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xslb9" event={"ID":"b99dd331-1d6d-434c-9b02-1b7694955006","Type":"ContainerStarted","Data":"3aba3250bfb20a60bc07e992dab181daa77b3a0019b1db9e6ad7c59a05535b39"}
Oct 30 23:06:14.986068 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:14.986037    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs\") pod \"network-metrics-daemon-pln6h\" (UID: \"58ec1adc-599b-4796-b12e-1cc85a834b15\") " pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:06:14.986241 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:14.986082    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:14.986241 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:14.986178    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 23:06:14.986241 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:14.986187    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:06:14.986241 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:14.986227    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret podName:2e942185-68e3-4d57-9f42-49d2965b41e4 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:46.986214684 +0000 UTC m=+65.437833442 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret") pod "global-pull-secret-syncer-cthfm" (UID: "2e942185-68e3-4d57-9f42-49d2965b41e4") : object "kube-system"/"original-pull-secret" not registered
Oct 30 23:06:14.986398 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:14.986246    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs podName:58ec1adc-599b-4796-b12e-1cc85a834b15 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:46.986233424 +0000 UTC m=+65.437852200 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs") pod "network-metrics-daemon-pln6h" (UID: "58ec1adc-599b-4796-b12e-1cc85a834b15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 23:06:14.996197 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:14.996176    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:14.996336 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:14.996320    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:06:14.996453 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:14.996435    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:06:15.049698 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:15.049677    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 23:06:15.086968 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:15.086946    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46ww\" (UniqueName: \"kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww\") pod \"network-check-target-qvbgk\" (UID: \"70a51653-2167-4e11-bf04-81f5b10c9c9d\") " pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:06:15.091805 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:15.091781    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 23:06:15.099794 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:15.099769    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tw7s4\""
Oct 30 23:06:15.100025 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:15.100006    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbzzd\""
Oct 30 23:06:15.100528 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:15.100503    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 23:06:15.100952 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:15.100912    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-trr47\""
Oct 30 23:06:15.101070 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:15.100958    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 23:06:15.101185 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:15.101168    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q46ww\" (UniqueName: \"kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww\") pod \"network-check-target-qvbgk\" (UID: \"70a51653-2167-4e11-bf04-81f5b10c9c9d\") " pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:06:15.188213 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:15.188125    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:15.188697 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:15.188285    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:06:15.188697 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:15.188347    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls podName:4e4bb115-b334-423e-86f1-1e1abb499b5d nodeName:}" failed. No retries permitted until 2026-10-30 23:06:17.188330621 +0000 UTC m=+35.639949383 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls") pod "dns-default-8qhk5" (UID: "4e4bb115-b334-423e-86f1-1e1abb499b5d") : secret "dns-default-metrics-tls" not found
Oct 30 23:06:15.315075 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:15.315040    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:06:15.688670 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:15.688582    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qvbgk"]
Oct 30 23:06:15.691949 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:06:15.691916    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a51653_2167_4e11_bf04_81f5b10c9c9d.slice/crio-b07b7204cd39e9c41de4b968110698ec83ed0e02861ef87ee4c0d55d56a2fc13 WatchSource:0}: Error finding container b07b7204cd39e9c41de4b968110698ec83ed0e02861ef87ee4c0d55d56a2fc13: Status 404 returned error can't find the container with id b07b7204cd39e9c41de4b968110698ec83ed0e02861ef87ee4c0d55d56a2fc13
Oct 30 23:06:16.228345 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:16.228315    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qvbgk" event={"ID":"70a51653-2167-4e11-bf04-81f5b10c9c9d","Type":"ContainerStarted","Data":"b07b7204cd39e9c41de4b968110698ec83ed0e02861ef87ee4c0d55d56a2fc13"}
Oct 30 23:06:16.951146 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:16.951068    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-tcf86"]
Oct 30 23:06:16.981633 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:16.981591    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:17.086201 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.086119    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tcf86"]
Oct 30 23:06:17.102157 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.102125    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:17.102283 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.102177    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzppf\" (UniqueName: \"kubernetes.io/projected/4f785b83-23af-45c1-975c-715bdebea140-kube-api-access-nzppf\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:17.102283 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.102204    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f785b83-23af-45c1-975c-715bdebea140-crio-socket\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:17.102283 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.102253    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f785b83-23af-45c1-975c-715bdebea140-data-volume\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:17.102283 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.102268    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f785b83-23af-45c1-975c-715bdebea140-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:17.135453 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.135372    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 23:06:17.135453 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.135374    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 23:06:17.135453 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.135454    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 23:06:17.148979 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.148953    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 23:06:17.149104 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.149010    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v482q\""
Oct 30 23:06:17.202330 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.202262    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-7v4kl-bjfqg"]
Oct 30 23:06:17.206740 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.203475    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f785b83-23af-45c1-975c-715bdebea140-crio-socket\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:17.206740 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.203570    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f785b83-23af-45c1-975c-715bdebea140-data-volume\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:17.206740 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.203623    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f785b83-23af-45c1-975c-715bdebea140-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:17.206740 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.203697    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:17.206740 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.203727    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:17.206740 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.203773    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzppf\" (UniqueName: \"kubernetes.io/projected/4f785b83-23af-45c1-975c-715bdebea140-kube-api-access-nzppf\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:17.206740 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:17.203974    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:06:17.206740 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:17.204048    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls podName:4e4bb115-b334-423e-86f1-1e1abb499b5d nodeName:}" failed. No retries permitted until 2026-10-30 23:06:21.204025187 +0000 UTC m=+39.655643947 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls") pod "dns-default-8qhk5" (UID: "4e4bb115-b334-423e-86f1-1e1abb499b5d") : secret "dns-default-metrics-tls" not found
Oct 30 23:06:17.206740 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.204232    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f785b83-23af-45c1-975c-715bdebea140-crio-socket\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:17.206740 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:17.204373    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 23:06:17.206740 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:17.204468    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4f785b83-23af-45c1-975c-715bdebea140-kube-rbac-proxy-cm podName:4f785b83-23af-45c1-975c-715bdebea140 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:17.704449898 +0000 UTC m=+36.156068657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4f785b83-23af-45c1-975c-715bdebea140-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tcf86" (UID: "4f785b83-23af-45c1-975c-715bdebea140") : configmap "kube-rbac-proxy" not found
Oct 30 23:06:17.206740 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:17.204508    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:06:17.206740 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:17.204563    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls podName:4f785b83-23af-45c1-975c-715bdebea140 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:17.704548323 +0000 UTC m=+36.156167095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tcf86" (UID: "4f785b83-23af-45c1-975c-715bdebea140") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:06:17.206740 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.205824    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f785b83-23af-45c1-975c-715bdebea140-data-volume\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:17.222930 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.222906    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:17.270502 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.270474    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7v4kl-bjfqg"]
Oct 30 23:06:17.281511 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.281485    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzppf\" (UniqueName: \"kubernetes.io/projected/4f785b83-23af-45c1-975c-715bdebea140-kube-api-access-nzppf\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:17.282177 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.282155    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 23:06:17.282328 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.282301    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6m9zb\""
Oct 30 23:06:17.304125 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.304098    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9bd99d0d-1f0f-4926-b835-880a64193ded-archives-path\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:17.304240 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.304139    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-service-ca-bundle\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:17.304240 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.304172    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-trusted-ca-bundle\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:17.304240 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.304204    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2kbb\" (UniqueName: \"kubernetes.io/projected/9bd99d0d-1f0f-4926-b835-880a64193ded-kube-api-access-k2kbb\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:17.367632 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.367580    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 23:06:17.405206 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.405175    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9bd99d0d-1f0f-4926-b835-880a64193ded-archives-path\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:17.405206 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.405213    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-service-ca-bundle\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:17.405432 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.405252    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-trusted-ca-bundle\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:17.405432 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.405279    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2kbb\" (UniqueName: \"kubernetes.io/projected/9bd99d0d-1f0f-4926-b835-880a64193ded-kube-api-access-k2kbb\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:17.405645 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.405622    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9bd99d0d-1f0f-4926-b835-880a64193ded-archives-path\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:17.405891 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.405874    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-service-ca-bundle\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:17.406087 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.406070    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-trusted-ca-bundle\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:17.461378 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.461326    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-m896h"]
Oct 30 23:06:17.467604 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.467580    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2kbb\" (UniqueName: \"kubernetes.io/projected/9bd99d0d-1f0f-4926-b835-880a64193ded-kube-api-access-k2kbb\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:17.487829 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.487810    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m896h"
Oct 30 23:06:17.521494 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.521463    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-m896h"]
Oct 30 23:06:17.540952 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.540932    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-w9j6t\""
Oct 30 23:06:17.550003 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.549979    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:17.589292 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.589275    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 23:06:17.589474 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.589454    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 23:06:17.607464 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.607440    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngj6h\" (UniqueName: \"kubernetes.io/projected/cdee91d4-262d-4142-9340-4243df4586b6-kube-api-access-ngj6h\") pod \"migrator-66df5d496b-m896h\" (UID: \"cdee91d4-262d-4142-9340-4243df4586b6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m896h"
Oct 30 23:06:17.708825 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.708797    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f785b83-23af-45c1-975c-715bdebea140-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:17.708924 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.708864    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngj6h\" (UniqueName: \"kubernetes.io/projected/cdee91d4-262d-4142-9340-4243df4586b6-kube-api-access-ngj6h\") pod \"migrator-66df5d496b-m896h\" (UID: \"cdee91d4-262d-4142-9340-4243df4586b6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m896h"
Oct 30 23:06:17.708924 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.708893    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:17.709025 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:17.708999    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:06:17.709060 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:17.709046    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls podName:4f785b83-23af-45c1-975c-715bdebea140 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:18.709030634 +0000 UTC m=+37.160649401 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tcf86" (UID: "4f785b83-23af-45c1-975c-715bdebea140") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:06:17.709104 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:17.709091    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 23:06:17.709136 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:17.709123    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4f785b83-23af-45c1-975c-715bdebea140-kube-rbac-proxy-cm podName:4f785b83-23af-45c1-975c-715bdebea140 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:18.709112441 +0000 UTC m=+37.160731199 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4f785b83-23af-45c1-975c-715bdebea140-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tcf86" (UID: "4f785b83-23af-45c1-975c-715bdebea140") : configmap "kube-rbac-proxy" not found
Oct 30 23:06:17.766804 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.766768    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngj6h\" (UniqueName: \"kubernetes.io/projected/cdee91d4-262d-4142-9340-4243df4586b6-kube-api-access-ngj6h\") pod \"migrator-66df5d496b-m896h\" (UID: \"cdee91d4-262d-4142-9340-4243df4586b6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m896h"
Oct 30 23:06:17.815475 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:17.815448    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m896h"
Oct 30 23:06:18.039060 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:18.038984    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7v4kl-bjfqg"]
Oct 30 23:06:18.042422 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:06:18.042392    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bd99d0d_1f0f_4926_b835_880a64193ded.slice/crio-f3d048a7dac8692c140c08e6cdf2c6de069f73e17c8625044406e1ad267865a0 WatchSource:0}: Error finding container f3d048a7dac8692c140c08e6cdf2c6de069f73e17c8625044406e1ad267865a0: Status 404 returned error can't find the container with id f3d048a7dac8692c140c08e6cdf2c6de069f73e17c8625044406e1ad267865a0
Oct 30 23:06:18.199027 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:18.199002    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-m896h"]
Oct 30 23:06:18.202403 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:06:18.202374    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdee91d4_262d_4142_9340_4243df4586b6.slice/crio-9f1d8828fff8ad25304de019af726541636d3be49ad97ab157a702fee3798b60 WatchSource:0}: Error finding container 9f1d8828fff8ad25304de019af726541636d3be49ad97ab157a702fee3798b60: Status 404 returned error can't find the container with id 9f1d8828fff8ad25304de019af726541636d3be49ad97ab157a702fee3798b60
Oct 30 23:06:18.235389 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:18.235358    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m896h" event={"ID":"cdee91d4-262d-4142-9340-4243df4586b6","Type":"ContainerStarted","Data":"9f1d8828fff8ad25304de019af726541636d3be49ad97ab157a702fee3798b60"}
Oct 30 23:06:18.236810 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:18.236784    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7v4kl-bjfqg" event={"ID":"9bd99d0d-1f0f-4926-b835-880a64193ded","Type":"ContainerStarted","Data":"f3d048a7dac8692c140c08e6cdf2c6de069f73e17c8625044406e1ad267865a0"}
Oct 30 23:06:18.238428 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:18.238400    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xslb9" event={"ID":"b99dd331-1d6d-434c-9b02-1b7694955006","Type":"ContainerStarted","Data":"acef40eb52198fdf8b7378da4900eaddfc33f9f069a67b811ca30ea71618461c"}
Oct 30 23:06:18.355169 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:18.355123    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xslb9" podStartSLOduration=1.837535558 podStartE2EDuration="5.355106961s" totalImagesPullingTime="3.517571403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:18.354352513 +0000 UTC m=+36.805971282" watchObservedRunningTime="2026-10-30 23:06:18.355106961 +0000 UTC m=+36.806725731"
Oct 30 23:06:18.717494 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:18.717412    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f785b83-23af-45c1-975c-715bdebea140-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:18.717661 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:18.717496    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:18.717661 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:18.717539    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 23:06:18.717661 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:18.717612    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4f785b83-23af-45c1-975c-715bdebea140-kube-rbac-proxy-cm podName:4f785b83-23af-45c1-975c-715bdebea140 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:20.717584236 +0000 UTC m=+39.169202982 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4f785b83-23af-45c1-975c-715bdebea140-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tcf86" (UID: "4f785b83-23af-45c1-975c-715bdebea140") : configmap "kube-rbac-proxy" not found
Oct 30 23:06:18.717801 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:18.717657    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:06:18.717801 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:18.717713    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls podName:4f785b83-23af-45c1-975c-715bdebea140 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:20.717700996 +0000 UTC m=+39.169319740 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tcf86" (UID: "4f785b83-23af-45c1-975c-715bdebea140") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:06:19.054682 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.054652    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-9jlqk"]
Oct 30 23:06:19.070807 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.070765    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk"
Oct 30 23:06:19.116577 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.116543    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-9jlqk"]
Oct 30 23:06:19.124010 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.123979    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-c9hs4\""
Oct 30 23:06:19.124237 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.124214    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 23:06:19.124557 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.124536    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 23:06:19.124681 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.124627    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 23:06:19.124729 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.124695    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 23:06:19.223557 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.223518    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/06742bba-beca-46c9-8614-2fa5973c4864-signing-cabundle\") pod \"service-ca-5fffc97bdf-9jlqk\" (UID: \"06742bba-beca-46c9-8614-2fa5973c4864\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk"
Oct 30 23:06:19.223739 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.223569    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/06742bba-beca-46c9-8614-2fa5973c4864-signing-key\") pod \"service-ca-5fffc97bdf-9jlqk\" (UID: \"06742bba-beca-46c9-8614-2fa5973c4864\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk"
Oct 30 23:06:19.223739 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.223637    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqcxx\" (UniqueName: \"kubernetes.io/projected/06742bba-beca-46c9-8614-2fa5973c4864-kube-api-access-rqcxx\") pod \"service-ca-5fffc97bdf-9jlqk\" (UID: \"06742bba-beca-46c9-8614-2fa5973c4864\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk"
Oct 30 23:06:19.324213 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.324134    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/06742bba-beca-46c9-8614-2fa5973c4864-signing-cabundle\") pod \"service-ca-5fffc97bdf-9jlqk\" (UID: \"06742bba-beca-46c9-8614-2fa5973c4864\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk"
Oct 30 23:06:19.324213 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.324171    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/06742bba-beca-46c9-8614-2fa5973c4864-signing-key\") pod \"service-ca-5fffc97bdf-9jlqk\" (UID: \"06742bba-beca-46c9-8614-2fa5973c4864\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk"
Oct 30 23:06:19.324213 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.324207    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqcxx\" (UniqueName: \"kubernetes.io/projected/06742bba-beca-46c9-8614-2fa5973c4864-kube-api-access-rqcxx\") pod \"service-ca-5fffc97bdf-9jlqk\" (UID: \"06742bba-beca-46c9-8614-2fa5973c4864\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk"
Oct 30 23:06:19.324982 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.324946    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/06742bba-beca-46c9-8614-2fa5973c4864-signing-cabundle\") pod \"service-ca-5fffc97bdf-9jlqk\" (UID: \"06742bba-beca-46c9-8614-2fa5973c4864\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk"
Oct 30 23:06:19.330485 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.330441    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/06742bba-beca-46c9-8614-2fa5973c4864-signing-key\") pod \"service-ca-5fffc97bdf-9jlqk\" (UID: \"06742bba-beca-46c9-8614-2fa5973c4864\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk"
Oct 30 23:06:19.382208 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.382182    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqcxx\" (UniqueName: \"kubernetes.io/projected/06742bba-beca-46c9-8614-2fa5973c4864-kube-api-access-rqcxx\") pod \"service-ca-5fffc97bdf-9jlqk\" (UID: \"06742bba-beca-46c9-8614-2fa5973c4864\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk"
Oct 30 23:06:19.410223 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.410057    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk"
Oct 30 23:06:19.819037 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:19.819001    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-9jlqk"]
Oct 30 23:06:19.824561 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:06:19.824535    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06742bba_beca_46c9_8614_2fa5973c4864.slice/crio-1636f9a77eb115dae1851b59480a76d79e9d59ee8225089d11ab34c04ecda05d WatchSource:0}: Error finding container 1636f9a77eb115dae1851b59480a76d79e9d59ee8225089d11ab34c04ecda05d: Status 404 returned error can't find the container with id 1636f9a77eb115dae1851b59480a76d79e9d59ee8225089d11ab34c04ecda05d
Oct 30 23:06:20.245357 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:20.245280    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk" event={"ID":"06742bba-beca-46c9-8614-2fa5973c4864","Type":"ContainerStarted","Data":"1636f9a77eb115dae1851b59480a76d79e9d59ee8225089d11ab34c04ecda05d"}
Oct 30 23:06:20.736545 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:20.736516    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:20.736932 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:20.736636    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f785b83-23af-45c1-975c-715bdebea140-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:20.736932 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:20.736678    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:06:20.736932 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:20.736705    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 23:06:20.736932 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:20.736739    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls podName:4f785b83-23af-45c1-975c-715bdebea140 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:24.736724545 +0000 UTC m=+43.188343294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tcf86" (UID: "4f785b83-23af-45c1-975c-715bdebea140") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:06:20.736932 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:20.736753    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4f785b83-23af-45c1-975c-715bdebea140-kube-rbac-proxy-cm podName:4f785b83-23af-45c1-975c-715bdebea140 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:24.736747336 +0000 UTC m=+43.188366080 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4f785b83-23af-45c1-975c-715bdebea140-kube-rbac-proxy-cm") pod "insights-runtime-extractor-tcf86" (UID: "4f785b83-23af-45c1-975c-715bdebea140") : configmap "kube-rbac-proxy" not found
Oct 30 23:06:21.241569 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:21.241529    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:21.241777 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:21.241695    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:06:21.241777 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:21.241762    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls podName:4e4bb115-b334-423e-86f1-1e1abb499b5d nodeName:}" failed. No retries permitted until 2026-10-30 23:06:29.24174541 +0000 UTC m=+47.693364159 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls") pod "dns-default-8qhk5" (UID: "4e4bb115-b334-423e-86f1-1e1abb499b5d") : secret "dns-default-metrics-tls" not found
Oct 30 23:06:21.249638 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:21.249586    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qvbgk" event={"ID":"70a51653-2167-4e11-bf04-81f5b10c9c9d","Type":"ContainerStarted","Data":"271cc7f9a3a02b3fecab6384d173cc35e68479961b2c12c0776db41511d91e21"}
Oct 30 23:06:21.249791 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:21.249724    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:06:21.251328 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:21.251307    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m896h" event={"ID":"cdee91d4-262d-4142-9340-4243df4586b6","Type":"ContainerStarted","Data":"8f4a1b49a59f6dc5fc4f313c5f29727348d61c0549f549a210c7758e209486ea"}
Oct 30 23:06:21.252942 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:21.252919    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7v4kl-bjfqg" event={"ID":"9bd99d0d-1f0f-4926-b835-880a64193ded","Type":"ContainerStarted","Data":"f20253b85443155ced8d865281bbea235538ce29ff6c844d3d4762a7ddbdcd65"}
Oct 30 23:06:21.364528 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:21.364462    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qvbgk" podStartSLOduration=33.639925703 podStartE2EDuration="38.364449323s" totalImagesPullingTime="4.72452362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:21.364008839 +0000 UTC m=+39.815627609" watchObservedRunningTime="2026-10-30 23:06:21.364449323 +0000 UTC m=+39.816068089"
Oct 30 23:06:21.482365 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:21.482316    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-7v4kl-bjfqg" podStartSLOduration=1.682285694 podStartE2EDuration="4.482301252s" totalImagesPullingTime="2.800015558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:21.482169966 +0000 UTC m=+39.933788734" watchObservedRunningTime="2026-10-30 23:06:21.482301252 +0000 UTC m=+39.933920019"
Oct 30 23:06:21.599466 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:21.599413    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m896h" podStartSLOduration=2.3972935140000002 podStartE2EDuration="4.599399409s" totalImagesPullingTime="2.202105895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:21.59888941 +0000 UTC m=+40.050508178" watchObservedRunningTime="2026-10-30 23:06:21.599399409 +0000 UTC m=+40.051018175"
Oct 30 23:06:23.259379 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:23.259225    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk" event={"ID":"06742bba-beca-46c9-8614-2fa5973c4864","Type":"ContainerStarted","Data":"ec6c7a99ee9f4895fa76d2949a20bec0e6e75b334685eedd247a6b6862d0c3ce"}
Oct 30 23:06:23.375222 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:23.375171    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk" podStartSLOduration=1.3722509 podStartE2EDuration="4.375156384s" totalImagesPullingTime="3.002905484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:06:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:23.374499778 +0000 UTC m=+41.826118546" watchObservedRunningTime="2026-10-30 23:06:23.375156384 +0000 UTC m=+41.826775152"
Oct 30 23:06:24.546370 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:24.546338    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gldcx_108c7917-29b6-47bc-a590-35fd94c3de19/dns-node-resolver/0.log"
Oct 30 23:06:24.768275 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:24.768187    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f785b83-23af-45c1-975c-715bdebea140-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:24.768275 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:24.768246    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:24.768490 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:24.768342    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 23:06:24.768490 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:24.768393    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls podName:4f785b83-23af-45c1-975c-715bdebea140 nodeName:}" failed. No retries permitted until 2026-10-30 23:06:32.768381041 +0000 UTC m=+51.219999800 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls") pod "insights-runtime-extractor-tcf86" (UID: "4f785b83-23af-45c1-975c-715bdebea140") : secret "insights-runtime-extractor-tls" not found
Oct 30 23:06:24.768875 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:24.768853    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f785b83-23af-45c1-975c-715bdebea140-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:25.547817 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:25.547785    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2g2gr_df39047e-a25e-4661-a34e-a6bca0f344f2/node-ca/0.log"
Oct 30 23:06:26.547111 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:26.547085    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xslb9_b99dd331-1d6d-434c-9b02-1b7694955006/serve-healthcheck-canary/0.log"
Oct 30 23:06:26.746953 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:26.746926    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-m896h_cdee91d4-262d-4142-9340-4243df4586b6/migrator/0.log"
Oct 30 23:06:29.299793 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:29.299759    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:29.300206 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:29.299880    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 23:06:29.300206 ip-10-0-130-205 kubenswrapper[2573]: E1030 23:06:29.299950    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls podName:4e4bb115-b334-423e-86f1-1e1abb499b5d nodeName:}" failed. No retries permitted until 2026-10-30 23:06:45.29992576 +0000 UTC m=+63.751544505 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls") pod "dns-default-8qhk5" (UID: "4e4bb115-b334-423e-86f1-1e1abb499b5d") : secret "dns-default-metrics-tls" not found
Oct 30 23:06:32.829437 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:32.829401    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:32.833259 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:32.833231    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:32.944763 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:32.944726    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:33.316140 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:33.316107    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-tcf86"]
Oct 30 23:06:33.319152 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:06:33.319125    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f785b83_23af_45c1_975c_715bdebea140.slice/crio-995c387048a10c8ceea3500e3253ea660cf67ace643da0c2aded683099e26795 WatchSource:0}: Error finding container 995c387048a10c8ceea3500e3253ea660cf67ace643da0c2aded683099e26795: Status 404 returned error can't find the container with id 995c387048a10c8ceea3500e3253ea660cf67ace643da0c2aded683099e26795
Oct 30 23:06:34.287460 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:34.287418    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tcf86" event={"ID":"4f785b83-23af-45c1-975c-715bdebea140","Type":"ContainerStarted","Data":"6224f79a33e4c2cf3c92f672eb34070484abb7c2b5feb8ac18dcccdcd7f5857e"}
Oct 30 23:06:34.287926 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:34.287527    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tcf86" event={"ID":"4f785b83-23af-45c1-975c-715bdebea140","Type":"ContainerStarted","Data":"995c387048a10c8ceea3500e3253ea660cf67ace643da0c2aded683099e26795"}
Oct 30 23:06:35.291801 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:35.291764    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tcf86" event={"ID":"4f785b83-23af-45c1-975c-715bdebea140","Type":"ContainerStarted","Data":"9f2b9c66a646af7216e9cf081c180188099dc3c81c363f1a1faf0291c1f3139e"}
Oct 30 23:06:37.307156 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.307123    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-tcf86" event={"ID":"4f785b83-23af-45c1-975c-715bdebea140","Type":"ContainerStarted","Data":"8716ce38829ae700009e4a27bc744397089dec06b41567e597161ee5f5f2ce92"}
Oct 30 23:06:37.432925 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.432865    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-tcf86" podStartSLOduration=18.603796617 podStartE2EDuration="21.432850605s" totalImagesPullingTime="2.829053988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:06:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:37.432017606 +0000 UTC m=+55.883636374" watchObservedRunningTime="2026-10-30 23:06:37.432850605 +0000 UTC m=+55.884469371"
Oct 30 23:06:37.610204 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.610130    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-9vzv5"]
Oct 30 23:06:37.611955 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.611928    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-9vzv5"
Oct 30 23:06:37.667659 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.667632    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhw29\" (UniqueName: \"kubernetes.io/projected/2cd84d6c-d204-4e07-b591-9930cc9062a5-kube-api-access-qhw29\") pod \"downloads-6dd789c8b7-9vzv5\" (UID: \"2cd84d6c-d204-4e07-b591-9930cc9062a5\") " pod="openshift-console/downloads-6dd789c8b7-9vzv5"
Oct 30 23:06:37.667776 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.667712    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cd84d6c-d204-4e07-b591-9930cc9062a5-tmp\") pod \"downloads-6dd789c8b7-9vzv5\" (UID: \"2cd84d6c-d204-4e07-b591-9930cc9062a5\") " pod="openshift-console/downloads-6dd789c8b7-9vzv5"
Oct 30 23:06:37.684825 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.684797    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-7sq9v\""
Oct 30 23:06:37.690916 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.690895    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 23:06:37.691008 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.690940    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 23:06:37.701266 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.701242    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-9vzv5"]
Oct 30 23:06:37.756737 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.756711    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f5789fcdd-txvlr"]
Oct 30 23:06:37.758350 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.758332    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.768870 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.768849    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-installation-pull-secrets\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.768960 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.768878    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cd84d6c-d204-4e07-b591-9930cc9062a5-tmp\") pod \"downloads-6dd789c8b7-9vzv5\" (UID: \"2cd84d6c-d204-4e07-b591-9930cc9062a5\") " pod="openshift-console/downloads-6dd789c8b7-9vzv5"
Oct 30 23:06:37.768960 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.768899    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-trusted-ca\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.769053 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.768961    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-registry-tls\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.769053 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.768988    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-registry-certificates\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.769053 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.769007    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhw29\" (UniqueName: \"kubernetes.io/projected/2cd84d6c-d204-4e07-b591-9930cc9062a5-kube-api-access-qhw29\") pod \"downloads-6dd789c8b7-9vzv5\" (UID: \"2cd84d6c-d204-4e07-b591-9930cc9062a5\") " pod="openshift-console/downloads-6dd789c8b7-9vzv5"
Oct 30 23:06:37.769053 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.769046    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzt62\" (UniqueName: \"kubernetes.io/projected/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-kube-api-access-hzt62\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.769233 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.769131    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-bound-sa-token\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.769233 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.769178    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cd84d6c-d204-4e07-b591-9930cc9062a5-tmp\") pod \"downloads-6dd789c8b7-9vzv5\" (UID: \"2cd84d6c-d204-4e07-b591-9930cc9062a5\") " pod="openshift-console/downloads-6dd789c8b7-9vzv5"
Oct 30 23:06:37.769233 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.769198    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-ca-trust-extracted\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.769334 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.769240    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-image-registry-private-configuration\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.866124 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.866049    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 23:06:37.866124 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.866121    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 23:06:37.866396 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.866145    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 23:06:37.866396 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.866214    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cl4w9\""
Oct 30 23:06:37.866396 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.866314    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 23:06:37.867468 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.867448    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f5789fcdd-txvlr"]
Oct 30 23:06:37.868969 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.868950    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhw29\" (UniqueName: \"kubernetes.io/projected/2cd84d6c-d204-4e07-b591-9930cc9062a5-kube-api-access-qhw29\") pod \"downloads-6dd789c8b7-9vzv5\" (UID: \"2cd84d6c-d204-4e07-b591-9930cc9062a5\") " pod="openshift-console/downloads-6dd789c8b7-9vzv5"
Oct 30 23:06:37.869818 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.869788    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-ca-trust-extracted\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.869905 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.869829    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-image-registry-private-configuration\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.869905 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.869856    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-installation-pull-secrets\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.869905 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.869886    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-trusted-ca\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.870064 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.869913    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-registry-tls\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.870064 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.869939    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-registry-certificates\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.870064 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.869965    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzt62\" (UniqueName: \"kubernetes.io/projected/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-kube-api-access-hzt62\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.870064 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.870013    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-bound-sa-token\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.870262 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.870225    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-ca-trust-extracted\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.871240 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.871213    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-trusted-ca\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.871336 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.871218    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-registry-certificates\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.872266 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.872242    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-image-registry-private-configuration\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.872353 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.872287    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-installation-pull-secrets\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.872797 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.872783    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-registry-tls\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.920714 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.920690    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-9vzv5"
Oct 30 23:06:37.928050 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.928029    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-bound-sa-token\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:37.928140 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:37.928094    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzt62\" (UniqueName: \"kubernetes.io/projected/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-kube-api-access-hzt62\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:38.068399 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:38.068365    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:38.343236 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:38.343205    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-9vzv5"]
Oct 30 23:06:38.345877 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:06:38.345853    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd84d6c_d204_4e07_b591_9930cc9062a5.slice/crio-8b83d6f660917e9f5f7b17c578365082c77ecf1744fd9a8adf73375b0bcbd39f WatchSource:0}: Error finding container 8b83d6f660917e9f5f7b17c578365082c77ecf1744fd9a8adf73375b0bcbd39f: Status 404 returned error can't find the container with id 8b83d6f660917e9f5f7b17c578365082c77ecf1744fd9a8adf73375b0bcbd39f
Oct 30 23:06:38.445987 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:38.445952    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f5789fcdd-txvlr"]
Oct 30 23:06:38.448405 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:06:38.448374    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda07ae84d_1e2a_4f1e_a763_e5363ccdd520.slice/crio-88a97cace2ee8aca50c00da9c49d90bcf2d4af5d848140b1cf2002340b83d516 WatchSource:0}: Error finding container 88a97cace2ee8aca50c00da9c49d90bcf2d4af5d848140b1cf2002340b83d516: Status 404 returned error can't find the container with id 88a97cace2ee8aca50c00da9c49d90bcf2d4af5d848140b1cf2002340b83d516
Oct 30 23:06:39.222041 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:39.222015    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:39.314724 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:39.314685    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-9vzv5" event={"ID":"2cd84d6c-d204-4e07-b591-9930cc9062a5","Type":"ContainerStarted","Data":"8b83d6f660917e9f5f7b17c578365082c77ecf1744fd9a8adf73375b0bcbd39f"}
Oct 30 23:06:39.316507 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:39.316419    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr" event={"ID":"a07ae84d-1e2a-4f1e-a763-e5363ccdd520","Type":"ContainerStarted","Data":"33da2dd4a2c7465ac708213802becabcb98512a8a4b181bbbfb84063ef13c117"}
Oct 30 23:06:39.316507 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:39.316471    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr" event={"ID":"a07ae84d-1e2a-4f1e-a763-e5363ccdd520","Type":"ContainerStarted","Data":"88a97cace2ee8aca50c00da9c49d90bcf2d4af5d848140b1cf2002340b83d516"}
Oct 30 23:06:39.316714 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:39.316565    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:39.470449 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:39.470390    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr" podStartSLOduration=2.470377613 podStartE2EDuration="2.470377613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:39.469793726 +0000 UTC m=+57.921412494" watchObservedRunningTime="2026-10-30 23:06:39.470377613 +0000 UTC m=+57.921996379"
Oct 30 23:06:45.327197 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:45.327157    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:45.329864 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:45.329840    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:45.646811 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:45.646734    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cqcdm\""
Oct 30 23:06:45.655702 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:45.655673    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:46.027888 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:46.027817    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8qhk5"]
Oct 30 23:06:46.031735 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:06:46.031700    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4bb115_b334_423e_86f1_1e1abb499b5d.slice/crio-93660373ddb05521fe81f84cff70a8d616799ede9b3dd690af5a7617ad1befaf WatchSource:0}: Error finding container 93660373ddb05521fe81f84cff70a8d616799ede9b3dd690af5a7617ad1befaf: Status 404 returned error can't find the container with id 93660373ddb05521fe81f84cff70a8d616799ede9b3dd690af5a7617ad1befaf
Oct 30 23:06:46.340300 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:46.340267    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8qhk5" event={"ID":"4e4bb115-b334-423e-86f1-1e1abb499b5d","Type":"ContainerStarted","Data":"93660373ddb05521fe81f84cff70a8d616799ede9b3dd690af5a7617ad1befaf"}
Oct 30 23:06:47.043829 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:47.043789    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs\") pod \"network-metrics-daemon-pln6h\" (UID: \"58ec1adc-599b-4796-b12e-1cc85a834b15\") " pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:06:47.044014 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:47.043854    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:47.146494 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:47.146436    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 23:06:47.146494 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:47.146467    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 23:06:47.158038 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:47.158005    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs\") pod \"network-metrics-daemon-pln6h\" (UID: \"58ec1adc-599b-4796-b12e-1cc85a834b15\") " pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:06:47.159201 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:47.159178    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:47.462194 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:47.461945    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tw7s4\""
Oct 30 23:06:47.470311 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:47.470288    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:47.512510 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:47.512483    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-trr47\""
Oct 30 23:06:47.521611 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:47.521571    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:06:47.847868 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:47.847837    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cthfm"]
Oct 30 23:06:47.851169 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:06:47.851132    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e942185_68e3_4d57_9f42_49d2965b41e4.slice/crio-029540db69f9838505973c4170867cfbcf771e1eaf79070f10248300de5452c1 WatchSource:0}: Error finding container 029540db69f9838505973c4170867cfbcf771e1eaf79070f10248300de5452c1: Status 404 returned error can't find the container with id 029540db69f9838505973c4170867cfbcf771e1eaf79070f10248300de5452c1
Oct 30 23:06:47.908328 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:47.908302    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-pln6h"]
Oct 30 23:06:47.913240 ip-10-0-130-205 kubenswrapper[2573]: W1030 23:06:47.913206    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ec1adc_599b_4796_b12e_1cc85a834b15.slice/crio-1633d8d790e534e1a6b63c58848a85702b22644518928f2c017cc79fe451df4d WatchSource:0}: Error finding container 1633d8d790e534e1a6b63c58848a85702b22644518928f2c017cc79fe451df4d: Status 404 returned error can't find the container with id 1633d8d790e534e1a6b63c58848a85702b22644518928f2c017cc79fe451df4d
Oct 30 23:06:48.348550 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:48.348437    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cthfm" event={"ID":"2e942185-68e3-4d57-9f42-49d2965b41e4","Type":"ContainerStarted","Data":"029540db69f9838505973c4170867cfbcf771e1eaf79070f10248300de5452c1"}
Oct 30 23:06:48.349904 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:48.349871    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pln6h" event={"ID":"58ec1adc-599b-4796-b12e-1cc85a834b15","Type":"ContainerStarted","Data":"1633d8d790e534e1a6b63c58848a85702b22644518928f2c017cc79fe451df4d"}
Oct 30 23:06:49.356463 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:49.356423    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8qhk5" event={"ID":"4e4bb115-b334-423e-86f1-1e1abb499b5d","Type":"ContainerStarted","Data":"e7afaa7b16559a2e3d61ea7bcae3e1b2806f0d70ea419e8028762e3b632f0458"}
Oct 30 23:06:49.356972 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:49.356492    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8qhk5" event={"ID":"4e4bb115-b334-423e-86f1-1e1abb499b5d","Type":"ContainerStarted","Data":"dfb44878973a71785e3bc1fa959b6607c67c228f36bf02d8e1f7a68c3c2e3c51"}
Oct 30 23:06:49.356972 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:49.356539    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:49.474216 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:49.474165    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-8qhk5" podStartSLOduration=34.061634891 podStartE2EDuration="36.474146699s" totalImagesPullingTime="2.412511808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:06:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:49.473087319 +0000 UTC m=+67.924706087" watchObservedRunningTime="2026-10-30 23:06:49.474146699 +0000 UTC m=+67.925765469"
Oct 30 23:06:50.364101 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:50.364067    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pln6h" event={"ID":"58ec1adc-599b-4796-b12e-1cc85a834b15","Type":"ContainerStarted","Data":"3b39d23340b6a0683755b88e95520d99fbc0dc4c93df521a7bb2c95dcac68c6a"}
Oct 30 23:06:50.364565 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:50.364126    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-pln6h" event={"ID":"58ec1adc-599b-4796-b12e-1cc85a834b15","Type":"ContainerStarted","Data":"ed08af3b649e079e12e09c0ea9108101ff4e9db924b6f217e45487b8166c513c"}
Oct 30 23:06:50.530672 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:50.530614    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-pln6h" podStartSLOduration=65.992518622 podStartE2EDuration="1m7.530581008s" totalImagesPullingTime="1.538062386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:05:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:50.483851984 +0000 UTC m=+68.935470778" watchObservedRunningTime="2026-10-30 23:06:50.530581008 +0000 UTC m=+68.982199778"
Oct 30 23:06:52.258383 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:52.258351    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:06:54.050417 ip-10-0-130-205 kubenswrapper[2573]: I1030 23:06:54.050365    2573 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:06:54.050530 ip-10-0-130-205 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 23:06:54.062103 ip-10-0-130-205 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 23:06:54.062329 ip-10-0-130-205 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 23:06:54.062390 ip-10-0-130-205 systemd[1]: kubelet.service: Consumed 3.501s CPU time.
Oct 30 23:06:54.075313 ip-10-0-130-205 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 23:06:54.181768 ip-10-0-130-205 kubenswrapper[5078]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:06:54.181768 ip-10-0-130-205 kubenswrapper[5078]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 23:06:54.181768 ip-10-0-130-205 kubenswrapper[5078]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:06:54.181768 ip-10-0-130-205 kubenswrapper[5078]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 23:06:54.184661 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184542    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:06:54.184661 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184661    5078 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:06:54.184661 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184664    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184669    5078 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184673    5078 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184676    5078 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184681    5078 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184683    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184687    5078 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184691    5078 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184696    5078 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184701    5078 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184706    5078 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184712    5078 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184716    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184721    5078 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184724    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184729    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184733    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184737    5078 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184741    5078 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184745    5078 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:06:54.184816 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184749    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184754    5078 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184757    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184766    5078 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184769    5078 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184772    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184774    5078 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184778    5078 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184780    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184783    5078 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184786    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184788    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184791    5078 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184794    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184796    5078 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184799    5078 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184801    5078 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184804    5078 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184806    5078 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184809    5078 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:06:54.185427 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184811    5078 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184814    5078 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184816    5078 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184820    5078 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184825    5078 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184828    5078 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184831    5078 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184833    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184836    5078 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184839    5078 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184841    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184844    5078 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184847    5078 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184849    5078 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184852    5078 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184856    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184859    5078 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184862    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184864    5078 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:06:54.186067 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184868    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184870    5078 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184873    5078 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184876    5078 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184878    5078 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184881    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184884    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184887    5078 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184890    5078 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184893    5078 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184897    5078 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184902    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184905    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184917    5078 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184922    5078 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184927    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184930    5078 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184934    5078 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184939    5078 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:06:54.186643 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184941    5078 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184944    5078 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184947    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184951    5078 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184955    5078 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184959    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184963    5078 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184967    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184971    5078 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184974    5078 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184977    5078 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184980    5078 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184983    5078 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184986    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184988    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184991    5078 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184994    5078 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.184997    5078 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.185000    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:06:54.187230 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.185002    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.185005    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.185007    5078 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.185010    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.185012    5078 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.185015    5078 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.185017    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.185969    5078 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.185978    5078 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.185981    5078 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.185985    5078 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.185989    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.185992    5078 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.185995    5078 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.185998    5078 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186001    5078 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186003    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186005    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186008    5078 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186010    5078 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:06:54.187904 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186013    5078 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186016    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186019    5078 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186022    5078 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186025    5078 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186028    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186031    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186034    5078 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186038    5078 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186042    5078 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186046    5078 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186050    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186054    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186058    5078 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186062    5078 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186066    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186070    5078 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186074    5078 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186078    5078 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186083    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:06:54.188514 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186088    5078 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186092    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186096    5078 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186100    5078 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186104    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186108    5078 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186112    5078 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186116    5078 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186122    5078 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186127    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186132    5078 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186137    5078 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186142    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186146    5078 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186149    5078 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186152    5078 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186156    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186165    5078 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186168    5078 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186171    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:06:54.189138 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186174    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186176    5078 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186179    5078 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186182    5078 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186184    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186187    5078 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186189    5078 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186192    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186194    5078 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186197    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186201    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186205    5078 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186210    5078 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186214    5078 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186218    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186222    5078 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186226    5078 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186230    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186238    5078 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186243    5078 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:06:54.189763 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186247    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186250    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186252    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186255    5078 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186257    5078 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186260    5078 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186262    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186265    5078 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186267    5078 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186269    5078 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186276    5078 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186279    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186282    5078 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186284    5078 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186287    5078 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186289    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186293    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186295    5078 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186298    5078 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186300    5078 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:06:54.190387 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186302    5078 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186306    5078 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186308    5078 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186311    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186313    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186316    5078 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186318    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186321    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186323    5078 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186326    5078 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186330    5078 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186332    5078 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186335    5078 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.186991    5078 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187004    5078 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187010    5078 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187017    5078 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187020    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187026    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187032    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:06:54.190998 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187036    5078 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187038    5078 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187042    5078 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187045    5078 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187049    5078 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187053    5078 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187056    5078 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187059    5078 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187062    5078 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187065    5078 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187067    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187070    5078 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187072    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187075    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187078    5078 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187080    5078 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187083    5078 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187085    5078 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187088    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187090    5078 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:06:54.191775 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187093    5078 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187096    5078 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187103    5078 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187107    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187112    5078 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187116    5078 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187119    5078 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187121    5078 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187125    5078 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187128    5078 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187130    5078 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187136    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187139    5078 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187141    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187144    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187147    5078 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187150    5078 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187152    5078 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187156    5078 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187161    5078 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:06:54.192324 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187166    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187170    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187174    5078 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187178    5078 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187183    5078 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187187    5078 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187192    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187196    5078 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187199    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187202    5078 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187204    5078 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187207    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187211    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187215    5078 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187221    5078 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187225    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187229    5078 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187233    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187235    5078 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187238    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:06:54.192912 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187241    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187243    5078 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187248    5078 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187253    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187257    5078 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187261    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187266    5078 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187271    5078 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187277    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187280    5078 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187282    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187285    5078 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187288    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187291    5078 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187294    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187297    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187299    5078 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187302    5078 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187304    5078 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:06:54.193535 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187307    5078 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187310    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187313    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187318    5078 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187322    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187326    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187331    5078 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187337    5078 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187341    5078 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187345    5078 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187350    5078 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187354    5078 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187357    5078 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187359    5078 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187362    5078 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187365    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187368    5078 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187371    5078 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187373    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.187376    5078 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:06:54.194173 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187474    5078 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187484    5078 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187497    5078 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187505    5078 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187511    5078 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187515    5078 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187530    5078 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187536    5078 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187541    5078 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187546    5078 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187552    5078 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187557    5078 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187561    5078 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187576    5078 flags.go:64] FLAG: --cgroup-root=""
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187579    5078 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187582    5078 flags.go:64] FLAG: --client-ca-file=""
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187585    5078 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187588    5078 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187608    5078 flags.go:64] FLAG: --cluster-domain=""
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187616    5078 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187620    5078 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187623    5078 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187626    5078 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187631    5078 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 23:06:54.194843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187634    5078 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187636    5078 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187640    5078 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187643    5078 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187647    5078 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187650    5078 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187653    5078 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187655    5078 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187663    5078 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187666    5078 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187669    5078 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187673    5078 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187678    5078 flags.go:64] FLAG: --enable-server="true"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187683    5078 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187691    5078 flags.go:64] FLAG: --event-burst="100"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187696    5078 flags.go:64] FLAG: --event-qps="50"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187702    5078 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187707    5078 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187712    5078 flags.go:64] FLAG: --eviction-hard=""
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187720    5078 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187724    5078 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187729    5078 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187734    5078 flags.go:64] FLAG: --eviction-soft=""
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187739    5078 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187744    5078 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 23:06:54.195614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187750    5078 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187755    5078 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187761    5078 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187766    5078 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187769    5078 flags.go:64] FLAG: --feature-gates=""
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187773    5078 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187776    5078 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187779    5078 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187782    5078 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187785    5078 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187788    5078 flags.go:64] FLAG: --help="false"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187794    5078 flags.go:64] FLAG: --hostname-override="ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187797    5078 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187800    5078 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187803    5078 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187806    5078 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187810    5078 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187813    5078 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187816    5078 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187820    5078 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187822    5078 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187825    5078 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187828    5078 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187831    5078 flags.go:64] FLAG: --kube-reserved=""
Oct 30 23:06:54.196397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187834    5078 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187837    5078 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187840    5078 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187843    5078 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187845    5078 flags.go:64] FLAG: --lock-file=""
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187848    5078 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187851    5078 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187854    5078 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187859    5078 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187862    5078 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187866    5078 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187869    5078 flags.go:64] FLAG: --logging-format="text"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187872    5078 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187875    5078 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187878    5078 flags.go:64] FLAG: --manifest-url=""
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187881    5078 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187887    5078 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187892    5078 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187898    5078 flags.go:64] FLAG: --max-pods="110"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187904    5078 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187907    5078 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187910    5078 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187913    5078 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187916    5078 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187919    5078 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 23:06:54.197117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187922    5078 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187930    5078 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187933    5078 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187937    5078 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187940    5078 flags.go:64] FLAG: --pod-cidr=""
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187943    5078 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187946    5078 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187949    5078 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187951    5078 flags.go:64] FLAG: --port="10250"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187954    5078 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187957    5078 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-02b7cc9bbc40822f3"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187960    5078 flags.go:64] FLAG: --qos-reserved=""
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187964    5078 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187969    5078 flags.go:64] FLAG: --register-node="true"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187974    5078 flags.go:64] FLAG: --register-with-taints=""
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187980    5078 flags.go:64] FLAG: --registry-burst="10"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187983    5078 flags.go:64] FLAG: --registry-qps="5"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187987    5078 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187990    5078 flags.go:64] FLAG: --reserved-memory=""
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187994    5078 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.187997    5078 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188000    5078 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188003    5078 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188006    5078 flags.go:64] FLAG: --runonce="false"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188009    5078 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188012    5078 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188016    5078 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 23:06:54.197988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188019    5078 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188023    5078 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188028    5078 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188032    5078 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188038    5078 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188043    5078 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188048    5078 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188053    5078 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188057    5078 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188063    5078 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188069    5078 flags.go:64] FLAG: --system-cgroups=""
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188074    5078 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188083    5078 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188087    5078 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188090    5078 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188094    5078 flags.go:64] FLAG: --tls-min-version=""
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188097    5078 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188100    5078 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188103    5078 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188106    5078 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188108    5078 flags.go:64] FLAG: --v="2"
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188113    5078 flags.go:64] FLAG: --version="false"
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188118    5078 flags.go:64] FLAG: --vmodule=""
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188122    5078 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188126    5078 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 23:06:54.198878 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188284    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188290    5078 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188294    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188296    5078 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188299    5078 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188301    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188307    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188311    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188316    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188320    5078 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188324    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188327    5078 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188330    5078 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188332    5078 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188335    5078 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188337    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188340    5078 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188344    5078 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188346    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188349    5078 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:06:54.199704 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188353    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188357    5078 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188361    5078 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188366    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188371    5078 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188375    5078 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188380    5078 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188385    5078 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188389    5078 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188395    5078 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188401    5078 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188405    5078 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188410    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188414    5078 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188419    5078 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188422    5078 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188424    5078 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188427    5078 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188432    5078 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188434    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:06:54.200466 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188437    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188440    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188442    5078 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188445    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188447    5078 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188450    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188452    5078 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188455    5078 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188458    5078 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188460    5078 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188464    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188467    5078 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188469    5078 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188472    5078 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188474    5078 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188477    5078 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188479    5078 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188482    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188484    5078 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188487    5078 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:06:54.201156 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188489    5078 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188493    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188496    5078 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188498    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188501    5078 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188503    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188506    5078 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188508    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188511    5078 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188514    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188518    5078 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188520    5078 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188523    5078 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188526    5078 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188529    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188532    5078 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188535    5078 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188537    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188540    5078 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188542    5078 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:06:54.201874 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188545    5078 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188547    5078 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188550    5078 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188553    5078 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188555    5078 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188558    5078 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188560    5078 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188563    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188565    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188569    5078 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188573    5078 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188577    5078 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188579    5078 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188583    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188585    5078 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188588    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188590    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188606    5078 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188610    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:06:54.202504 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188614    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:06:54.203323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188618    5078 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:06:54.203323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188622    5078 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:06:54.203323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188625    5078 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:06:54.203323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188627    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:06:54.203323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188630    5078 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:06:54.203323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188632    5078 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:06:54.203323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.188662    5078 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:06:54.203323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188702    5078 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:06:54.203323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188708    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:06:54.203323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188712    5078 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:06:54.203323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188718    5078 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:06:54.203323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188722    5078 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:06:54.203323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188726    5078 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:06:54.203323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188730    5078 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:06:54.203323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188734    5078 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:06:54.203323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188739    5078 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:06:54.203323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188745    5078 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188750    5078 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188755    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188759    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188765    5078 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188773    5078 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188777    5078 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188780    5078 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188783    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188785    5078 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188788    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188791    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188794    5078 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188796    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188798    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188801    5078 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188803    5078 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188806    5078 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188808    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:06:54.203968 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188810    5078 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188813    5078 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188815    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188819    5078 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188822    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188825    5078 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188828    5078 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188830    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188833    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188835    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188838    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188840    5078 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188843    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188845    5078 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188848    5078 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188851    5078 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188853    5078 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188856    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188858    5078 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188861    5078 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:06:54.204698 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188863    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188866    5078 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188868    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188870    5078 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188873    5078 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188875    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188878    5078 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188880    5078 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188883    5078 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188885    5078 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188887    5078 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188890    5078 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188893    5078 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188895    5078 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188898    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188900    5078 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188903    5078 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188905    5078 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188908    5078 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188910    5078 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:06:54.205543 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188914    5078 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188918    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188922    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188926    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188930    5078 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188932    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188935    5078 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188938    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188941    5078 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188943    5078 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188945    5078 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188948    5078 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188950    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188953    5078 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188955    5078 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188957    5078 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188959    5078 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188962    5078 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188964    5078 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188966    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:06:54.206436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188969    5078 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188971    5078 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188974    5078 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188976    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188978    5078 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188981    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188983    5078 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188986    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188988    5078 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188990    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188993    5078 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188995    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.188998    5078 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.189000    5078 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.189002    5078 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.189005    5078 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.189007    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.189010    5078 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.193426    5078 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.193440    5078 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 23:06:54.207440 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.193457    5078 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.193465    5078 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193534    5078 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193540    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193543    5078 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193546    5078 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193549    5078 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193552    5078 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193555    5078 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193558    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193562    5078 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193566    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193571    5078 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193576    5078 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193580    5078 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193585    5078 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193590    5078 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193616    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193619    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:06:54.208323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193622    5078 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193624    5078 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193627    5078 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193629    5078 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193632    5078 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193634    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193637    5078 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193639    5078 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193642    5078 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193644    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193646    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193649    5078 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193652    5078 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193655    5078 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193658    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193661    5078 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193664    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193666    5078 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193668    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193671    5078 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:06:54.209145 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193673    5078 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193676    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193678    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193681    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193685    5078 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193690    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193697    5078 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193704    5078 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193708    5078 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193711    5078 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193714    5078 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193717    5078 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193720    5078 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193722    5078 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193724    5078 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193727    5078 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193729    5078 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193732    5078 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193735    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:06:54.210015 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193739    5078 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193742    5078 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193747    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193751    5078 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193754    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193756    5078 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193759    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193762    5078 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193765    5078 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193767    5078 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193770    5078 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193772    5078 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193775    5078 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193777    5078 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193780    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193783    5078 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193785    5078 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193788    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193790    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193793    5078 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:06:54.210848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193795    5078 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193799    5078 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193803    5078 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193806    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193809    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193811    5078 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193814    5078 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193817    5078 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193819    5078 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193822    5078 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193824    5078 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193827    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193830    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193834    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193839    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193845    5078 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193849    5078 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193858    5078 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193861    5078 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:06:54.211759 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193864    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:06:54.212648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193867    5078 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:06:54.212648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193869    5078 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:06:54.212648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193872    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:06:54.212648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193874    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:06:54.212648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193877    5078 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:06:54.212648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193879    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:06:54.212648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193882    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:06:54.212648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193884    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:06:54.212648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193888    5078 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:06:54.212648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.193892    5078 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:06:54.212648 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.193923    5078 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:06:54.212648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194640    5078 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:06:54.212648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194654    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:06:54.212648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194660    5078 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:06:54.212648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194665    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:06:54.212648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194670    5078 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194676    5078 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194686    5078 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194691    5078 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194699    5078 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194703    5078 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194708    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194713    5078 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194717    5078 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194721    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194728    5078 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194734    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194740    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194745    5078 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194756    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194761    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194765    5078 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194769    5078 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194775    5078 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:06:54.213476 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194779    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194783    5078 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194788    5078 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194792    5078 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194796    5078 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194801    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194805    5078 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194809    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194818    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194822    5078 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194826    5078 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194831    5078 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194835    5078 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194839    5078 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194843    5078 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194848    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194853    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194858    5078 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194863    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194868    5078 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:06:54.214451 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194872    5078 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194881    5078 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194885    5078 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194889    5078 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194893    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194898    5078 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194902    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194908    5078 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194912    5078 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194916    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194921    5078 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194925    5078 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194931    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194936    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194945    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194949    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194954    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194957    5078 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194961    5078 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:06:54.215401 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194966    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194971    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194975    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194980    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194985    5078 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194989    5078 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.194993    5078 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195002    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195006    5078 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195011    5078 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195016    5078 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195021    5078 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195025    5078 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195029    5078 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195064    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195069    5078 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195074    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195078    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195082    5078 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195089    5078 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:06:54.216323 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195101    5078 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195106    5078 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195111    5078 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195116    5078 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195120    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195125    5078 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195130    5078 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195134    5078 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195139    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195143    5078 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195147    5078 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195151    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195156    5078 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195164    5078 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195170    5078 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195174    5078 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195178    5078 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195183    5078 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195187    5078 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195191    5078 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:06:54.217183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195195    5078 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:06:54.218050 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195199    5078 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:06:54.218050 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195206    5078 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:06:54.218050 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195211    5078 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:06:54.218050 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.195276    5078 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 23:06:54.218050 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195402    5078 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 23:06:54.218050 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195408    5078 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 23:06:54.218050 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195413    5078 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 23:06:54.218050 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195417    5078 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 23:06:54.218050 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195422    5078 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 23:06:54.218050 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195426    5078 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 23:06:54.218050 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195430    5078 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 23:06:54.218050 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195435    5078 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 23:06:54.218050 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195440    5078 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 23:06:54.218050 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195445    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 23:06:54.218050 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195449    5078 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 23:06:54.218050 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195453    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 23:06:54.218050 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195463    5078 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195467    5078 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195472    5078 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195477    5078 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195481    5078 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195485    5078 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195489    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195494    5078 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195498    5078 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195503    5078 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195509    5078 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195515    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195532    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195536    5078 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195541    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195545    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195549    5078 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195554    5078 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195559    5078 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 23:06:54.218815 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195563    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195568    5078 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195572    5078 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195576    5078 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195581    5078 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195585    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195610    5078 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195615    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195622    5078 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195626    5078 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195631    5078 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195635    5078 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195640    5078 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195644    5078 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195649    5078 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195653    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195658    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195663    5078 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195667    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195675    5078 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 23:06:54.219648 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195680    5078 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195684    5078 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195689    5078 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195693    5078 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195697    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195702    5078 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195706    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195711    5078 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195715    5078 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195719    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195723    5078 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195733    5078 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195737    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195742    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195746    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195750    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195755    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195758    5078 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195763    5078 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 23:06:54.220406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195767    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195772    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195776    5078 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195780    5078 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195791    5078 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195796    5078 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195800    5078 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195804    5078 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195808    5078 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195813    5078 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195817    5078 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195822    5078 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195827    5078 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195831    5078 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195835    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195839    5078 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195844    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195852    5078 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195857    5078 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195863    5078 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 23:06:54.221311 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195869    5078 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195873    5078 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195877    5078 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195882    5078 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195888    5078 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195892    5078 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195896    5078 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195900    5078 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195905    5078 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195913    5078 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195919    5078 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195923    5078 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195927    5078 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195932    5078 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195939    5078 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:54.195944    5078 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.196057    5078 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.197567    5078 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.197661    5078 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 23:06:54.222082 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.197880    5078 server.go:1014] "Starting client certificate rotation"
Oct 30 23:06:54.222862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.198033    5078 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 23:06:54.222862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.198109    5078 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:00:42 +0000 UTC" deadline="2028-04-09 21:35:50.815579298 +0000 UTC"
Oct 30 23:06:54.222862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.198149    5078 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12646h28m56.617433311s"
Oct 30 23:06:54.222862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.198412    5078 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:06:54.222862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.199815    5078 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 23:06:54.222862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.201198    5078 log.go:25] "Validated CRI v1 runtime API"
Oct 30 23:06:54.222862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.203162    5078 log.go:25] "Validated CRI v1 image API"
Oct 30 23:06:54.222862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.204117    5078 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 23:06:54.222862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.206766    5078 fs.go:135] Filesystem UUIDs: map[2a3a3824-3c87-4684-9503-30321fcf5616:/dev/nvme0n1p3 6197fd17-3730-4887-a18d-2d009f205b59:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 30 23:06:54.223312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.206799    5078 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/029540db69f9838505973c4170867cfbcf771e1eaf79070f10248300de5452c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/029540db69f9838505973c4170867cfbcf771e1eaf79070f10248300de5452c1/userdata/shm major:0 minor:306 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c0ebf017ce4c7526dddc3e8f9d6f9518b84b7ca69a5085f1e40912bbfd5fc03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c0ebf017ce4c7526dddc3e8f9d6f9518b84b7ca69a5085f1e40912bbfd5fc03/userdata/shm major:0 minor:62 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1633d8d790e534e1a6b63c58848a85702b22644518928f2c017cc79fe451df4d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1633d8d790e534e1a6b63c58848a85702b22644518928f2c017cc79fe451df4d/userdata/shm major:0 minor:308 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1636f9a77eb115dae1851b59480a76d79e9d59ee8225089d11ab34c04ecda05d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1636f9a77eb115dae1851b59480a76d79e9d59ee8225089d11ab34c04ecda05d/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3aba3250bfb20a60bc07e992dab181daa77b3a0019b1db9e6ad7c59a05535b39/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3aba3250bfb20a60bc07e992dab181daa77b3a0019b1db9e6ad7c59a05535b39/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/577bbe5f783c849e7733d86c2d6569bb0b9955764cc058b2a7c01a5fe3dda90c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/577bbe5f783c849e7733d86c2d6569bb0b9955764cc058b2a7c01a5fe3dda90c/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6619f18877b5bbac0bcd2088d544fb1fcb3733f5c8c3432c8f1bdec4bb752e3a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6619f18877b5bbac0bcd2088d544fb1fcb3733f5c8c3432c8f1bdec4bb752e3a/userdata/shm major:0 minor:85 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7711a49b12888c6f8c173ce50e08a979cc1d841d1d0218af21cc633134a29005/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7711a49b12888c6f8c173ce50e08a979cc1d841d1d0218af21cc633134a29005/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7de3c861ceb745ee60fac5d3355fbaf7a32b92c2101ba5aa447c370dbc1efe12/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7de3c861ceb745ee60fac5d3355fbaf7a32b92c2101ba5aa447c370dbc1efe12/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88a97cace2ee8aca50c00da9c49d90bcf2d4af5d848140b1cf2002340b83d516/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88a97cace2ee8aca50c00da9c49d90bcf2d4af5d848140b1cf2002340b83d516/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b83d6f660917e9f5f7b17c578365082c77ecf1744fd9a8adf73375b0bcbd39f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b83d6f660917e9f5f7b17c578365082c77ecf1744fd9a8adf73375b0bcbd39f/userdata/shm major:0 minor:283 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d753418031592ca311fa0bc6cb8bcc039543c68e5ec696435c1dcbfc260c320/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d753418031592ca311fa0bc6cb8bcc039543c68e5ec696435c1dcbfc260c320/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93660373ddb05521fe81f84cff70a8d616799ede9b3dd690af5a7617ad1befaf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93660373ddb05521fe81f84cff70a8d616799ede9b3dd690af5a7617ad1befaf/userdata/shm major:0 minor:300 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/995c387048a10c8ceea3500e3253ea660cf67ace643da0c2aded683099e26795/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/995c387048a10c8ceea3500e3253ea660cf67ace643da0c2aded683099e26795/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f1d8828fff8ad25304de019af726541636d3be49ad97ab157a702fee3798b60/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f1d8828fff8ad25304de019af726541636d3be49ad97ab157a702fee3798b60/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9f28f4ffabe36c7eb7853ab824409f4c93fc666101154ea2a9de302865ebb19a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9f28f4ffabe36c7eb7853ab824409f4c93fc666101154ea2a9de302865ebb19a/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a4e5e6789fe136c0f61393a6b0b46b93a51333e25d04454ec1d97192e25b68d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a4e5e6789fe136c0f61393a6b0b46b93a51333e25d04454ec1d97192e25b68d4/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b07b7204cd39e9c41de4b968110698ec83ed0e02861ef87ee4c0d55d56a2fc13/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b07b7204cd39e9c41de4b968110698ec83ed0e02861ef87ee4c0d55d56a2fc13/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e8d2abb44c70942aa77687775e2477d94c0dd840609a45ec23e29865c86c6d15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e8d2abb44c70942aa77687775e2477d94c0dd840609a45ec23e29865c86c6d15/userdata/shm major:0 minor:63 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f3d048a7dac8692c140c08e6cdf2c6de069f73e17c8625044406e1ad267865a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f3d048a7dac8692c140c08e6cdf2c6de069f73e17c8625044406e1ad267865a0/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f88938488131bbe50049327147b95a87b12c1e36a3b0941d9062a772190f1524/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f88938488131bbe50049327147b95a87b12c1e36a3b0941d9062a772190f1524/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06742bba-beca-46c9-8614-2fa5973c4864/volumes/kubernetes.io~projected/kube-api-access-rqcxx:{mountpoint:/var/lib/kubelet/pods/06742bba-beca-46c9-8614-2fa5973c4864/volumes/kubernetes.io~projected/kube-api-access-rqcxx major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06742bba-beca-46c9-8614-2fa5973c4864/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/06742bba-beca-46c9-8614-2fa5973c4864/volumes/kubernetes.io~secret/signing-key major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/108c7917-29b6-47bc-a590-35fd94c3de19/volumes/kubernetes.io~projected/kube-api-access-rqgxb:{mountpoint:/var/lib/kubelet/pods/108c7917-29b6-47bc-a590-35fd94c3de19/volumes/kubernetes.io~projected/kube-api-access-rqgxb major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1918245f-b465-485f-9603-04ebe4846085/volumes/kubernetes.io~projected/kube-api-access-pmhb8:{mountpoint:/var/lib/kubelet/pods/1918245f-b465-485f-9603-04ebe4846085/volumes/kubernetes.io~projected/kube-api-access-pmhb8 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22aabe7f-74b4-4cc7-b88b-1ec78aecc925/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/22aabe7f-74b4-4cc7-b88b-1ec78aecc925/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22aabe7f-74b4-4cc7-b88b-1ec78aecc925/volumes/kubernetes.io~projected/kube-api-access-z5p98:{mountpoint:/var/lib/kubelet/pods/22aabe7f-74b4-4cc7-b88b-1ec78aecc925/volumes/kubernetes.io~projected/kube-api-access-z5p98 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22aabe7f-74b4-4cc7-b88b-1ec78aecc925/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/22aabe7f-74b4-4cc7-b88b-1ec78aecc925/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf/volumes/kubernetes.io~projected/kube-api-access-9mdsg:{mountpoint:/var/lib/kubelet/pods/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf/volumes/kubernetes.io~projected/kube-api-access-9mdsg major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2cd84d6c-d204-4e07-b591-9930cc9062a5/volumes/kubernetes.io~projected/kube-api-access-qhw29:{mountpoint:/var/lib/kubelet/pods/2cd84d6c-d204-4e07-b591-9930cc9062a5/volumes/kubernetes.io~projected/kube-api-access-qhw29 major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e942185-68e3-4d57-9f42-49d2965b41e4/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2e942185-68e3-4d57-9f42-49d2965b41e4/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e4bb115-b334-423e-86f1-1e1abb499b5d/volumes/kubernetes.io~projected/kube-api-access-l2fzc:{mountpoint:/var/lib/kubelet/pods/4e4bb115-b334-423e-86f1-1e1abb499b5d/volumes/kubernetes.io~projected/kube-api-access-l2fzc major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e4bb115-b334-423e-86f1-1e1abb499b5d/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/4e4bb115-b334-423e-86f1-1e1abb499b5d/volumes/kubernetes.io~secret/metrics-tls major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f785b83-23af-45c1-975c-715bdebea140/volumes/kubernetes.io~projected/kube-api-access-nzppf:{mountpoint:/var/lib/kubelet/pods/4f785b83-23af-45c1-975c-715bdebea140/volumes/kubernetes.io~projected/kube-api-access-nzppf major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4f785b83-23af-45c1-975c-715bdebea140/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/4f785b83-23af-45c1-975c-715bdebea140/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58ec1adc-599b-4796-b12e-1cc85a834b15/volumes/kubernetes.io~projected/kube-api-access-mq95j:{mountpoint:/var/lib/kubelet/pods/58ec1adc-599b-4796-b12e-1cc85a834b15/volumes/kubernetes.io~projected/kube-api-access-mq95j major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58ec1adc-599b-4796-b12e-1cc85a834b15/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/58ec1adc-599b-4796-b12e-1cc85a834b15/volumes/kubernetes.io~secret/metrics-certs major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d5aef87-7d40-4aa2-958a-1fb73d427e3b/volumes/kubernetes.io~projected/kube-api-access-pb2fp:{mountpoint:/var/lib/kubelet/pods/6d5aef87-7d40-4aa2-958a-1fb73d427e3b/volumes/kubernetes.io~projected/kube-api-access-pb2fp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70a51653-2167-4e11-bf04-81f5b10c9c9d/volumes/kubernetes.io~projected/kube-api-access-q46ww:{mountpoint:/var/lib/kubelet/pods/70a51653-2167-4e11-bf04-81f5b10c9c9d/volumes/kubernetes.io~projected/kube-api-access-q46ww major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7e14c04d-b583-4f0a-840c-d493b2a2bbaf/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7e14c04d-b583-4f0a-840c-d493b2a2bbaf/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81b805c0-763e-4cd9-b6f5-af98767b298e/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/81b805c0-763e-4cd9-b6f5-af98767b298e/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81b805c0-763e-4cd9-b6f5-af98767b298e/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/81b805c0-763e-4cd9-b6f5-af98767b298e/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81b805c0-763e-4cd9-b6f5-af98767b298e/volumes/kubernetes.io~projected/kube-api-access-d6bjr:{mountpoint:/var/lib/kubelet/pods/81b805c0-763e-4cd9-b6f5-af98767b298e/volumes/kubernetes.io~projected/kube-api-access-d6bjr major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91deced5-f22e-4148-98da-6a232be165c3/volumes/kubernetes.io~projected/kube-api-access-kqblk:{mountpoint:/var/lib/kubelet/pods/91deced5-f22e-4148-98da-6a232be165c3/volumes/kubernetes.io~projected/kube-api-access-kqblk major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bd99d0d-1f0f-4926-b835-880a64193ded/volumes/kubernetes.io~projected/kube-api-access-k2kbb:{mountpoint:/var/lib/kubelet/pods/9bd99d0d-1f0f-4926-b835-880a64193ded/volumes/kubernetes.io~projected/kube-api-access-k2kbb major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a07ae84d-1e2a-4f1e-a763-e5363ccdd520/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/a07ae84d-1e2a-4f1e-a763-e5363ccdd520/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a07ae84d-1e2a-4f1e-a763-e5363ccdd520/volumes/kubernetes.io~projected/kube-api-access-hzt62:{mountpoint:/var/lib/kubelet/pods/a07ae84d-1e2a-4f1e-a763-e5363ccdd520/volumes/kubernetes.io~projected/kube-api-access-hzt62 major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a07ae84d-1e2a-4f1e-a763-e5363ccdd520/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/a07ae84d-1e2a-4f1e-a763-e5363ccdd520/volumes/kubernetes.io~projected/registry-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a07ae84d-1e2a-4f1e-a763-e5363ccdd520/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/a07ae84d-1e2a-4f1e-a763-e5363ccdd520/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a07ae84d-1e2a-4f1e-a763-e5363ccdd520/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/a07ae84d-1e2a-4f1e-a763-e5363ccdd520/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b99dd331-1d6d-434c-9b02-1b7694955006/volumes/kubernetes.io~projected/kube-api-access-5bm7q:{mountpoint:/var/lib/kubelet/pods/b99dd331-1d6d-434c-9b02-1b7694955006/volumes/kubernetes.io~projected/kube-api-access-5bm7q major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b99dd331-1d6d-434c-9b02-1b7694955006/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/b99dd331-1d6d-434c-9b02-1b7694955006/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cdee91d4-262d-4142-9340-4243df4586b6/volumes/kubernetes.io~projected/kube-api-access-ngj6h:{mountpoint:/var/lib/kubelet/pods/cdee91d4-262d-4142-9340-4243df4586b6/volumes/kubernetes.io~projected/kube-api-access-ngj6h major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df39047e-a25e-4661-a34e-a6bca0f344f2/volumes/kubernetes.io~projected/kube-api-access-p5zjd:{mountpoint:/var/lib/kubelet/pods/df39047e-a25e-4661-a34e-a6bca0f344f2/volumes/kubernetes.io~projected/kube-api-access-p5zjd major:0 minor:59 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/266b6647c1714e2e98a744fe1923d46df84818a5fc4b734e0d74f57b4cd7b714/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/82bba10af604ec1775f86bf1a146f1ea86495bdf6bf39ffa27e10fbcb35404ca/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-107:{mountpoint:/var/lib/containers/storage/overlay/2e553a7392bd11006cd0fa7bb43358e79cb6e38d9c25ca1141ddef1782209364/merged major:0 minor:107 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/9d1cf27aca5bf3eb2a636bd2b02ae8c346ba54dc60865e18fd1cf8dcd020dbab/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/4b5fc286723b6ff2c788bf062879cc359757f5212057437d7203a0139db942a7/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a3ae10b813aa5564332f7f66bec52356bd98048013e471897a1f83a26a8137aa/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/661e418c385fde5850a2c4c1230a1ecd114f41e03951f481d215db4bf88c4059/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/838150541df64459b874113a721cb80841e7e9dea735bae9a8a40d9eaafc0825/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/70dba82ec29ce7c65b08bc38d71111600792e921672201f5586b336110db99fc/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/57c955053ad276c7f3a57cab3cce6001398307a92df2c5bb646955fa96588731/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/a0cb753bb8300810300604cbe81dac631368d003242c3942b8554c792aeb32b7/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/482e6f3449e36da21f35a347a1b964665dd18b79b8e7ac777960e7184ff7ed3b/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/fc581a4ccfe1f7861b707da7c838d16baf0094e28723122200aff9ab0d9ea968/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/2b886e85bf01f486d919ac11b73be8ae8dc2d3c8f09e93b2d7d4dca6c9b30908/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/5be814f1ab8476d808df22d8cb86b36c0e1279bc5adefdb2145af5585cfd48f3/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/5c10c7a1e2e9899f34f4b7c9392921f510e4f118d8ffdfe0146e581fccd7a909/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/20d9c8dff56e64357d780ff9936200c644024d5fc7aa80ea9408b9e50e283539/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/4a3f659635e69b2858bb5b8a267e21d1515e69ef597b4a546223c4f5eab6f20c/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4e8c8099ef39c43f5d0e7f441db114a379210a24c6013189d4fecdc771de32b8/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/757115281fd78e7ea81e017ee73ecbcb70d496925b36d30145f2b2055592d64a/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/3a6225313d1e19c8b9c801348791ec517846deab4ce6cb96924ebee9fa81a39a/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/4bda1a7e7d2d000ea2c8255db9aa2d9c2cdde4234aba0da8c68bb98cb1d1c67f/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/557ceec0e5550a5c19d4a2b0c99a3877fce31fc3cb2bdbba2a71886126ebbd0c/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/126b728557fa4e15d5bc883d40fab47b42dc2058ef98eed42ccebbd202b46f49/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/6c185b815c79159b82e6a13eeb9da3799e814f3b568cf1079d7337ee6588892c/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/81e3d098d89d2658caea25d1d9cb6687b46dcbdfe139948a55473e99afc7c916/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/02eec6dd04bdaa3f5060c964a06ec564ef5e618eac2ba8cba3ed2fc0070ecd6d/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/3d58e9f8fa7ba2a9de6ec8694f5e156c541ecd95c259516bce7c698fd1b4f7dd/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/1462782722ee27e54c59deaa00ea243e31ed25fe5af1933ebadeb1e837834f6a/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/7f9ed474f5572c6107083de1b86a2f9204afed36cc475c2dd4b46b84b73770f6/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/12b91c26385c3e83b8aba52f0d9137d8c19d9176d29b17cd2da09f99f5945c9b/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/57743eaed7e01b3c49f11d450178cddab0b2217465797b525a625437ebfce7c0/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/fbc855c2aca04b0a5c17d02ffb516477ffde728a7ed1024716e011baa93e6fb8/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/418167f8c244f9fc877ce54182082cc269583168e8b83042096eb8469d860056/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/d30ea4b28a9dbe54c4bb5fe3da2378f6b2eaa4239e7875b75a72a3eac52bc9ed/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/84fe1a8405c3496642d6cb051b8aa27f690b4d78bcbb355da25f692c667c74e3/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/165adcf7b168f0cdb2d66aa2d0f9ffd1a92c916b93686040cd01f306617963fc/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/94ee1247ae4edf8d7ee3aca8e6e6ad93f25edbb66803e4ace75fb4e2feac29b7/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/a3369301679971dd7e4e74df6c430c0e021e4d0950be5701f5c1a7163d0c9d61/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-310:{mountpoint:/var/lib/containers/storage/overlay/acab5eb4b06d14cdc4f092aed166b9c9bc74aaebef73d05a2d6fe72b96ce641a/merged major:0 minor:310 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/ee8f8b317aeaa350d2479ff43a2c5c29f4e4e9f9b2b100721734ddfcb0f1bcff/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/9a77e9b94eb3432e1a764df45bf671bcda508565d3607996e69007ca7001a8af/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/097b83637c337f20a50f27de2f6a79a2a16408b603c1fd7f0a1270cf83fe1075/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/4463cba919d6ce059de7f7103f1342b6d59abe65a0992ede1aeb2df1bdebf434/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/d1b1b1ccb79227d137106d5e3e0734ae15a390461920cd72bf946f059bf6ff55/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-333:{mountpoint:/var/lib/containers/storage/overlay/26bd656ab6a5bc809ac8ac42805280619ea0126d34e22463a229d25649e756a1/merged major:0 minor:333 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/a507006cb7587633efdba7fa974c469904e6555d7805ceb47e127c9c5765e8bd/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/aec8f80dde496d85aca985b912c4247f1574fa33441db64b7a632ec6aee9813b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/c81f0cca9e7d252bae79bd53c9846e7abab6fe5a2a6917204a79a608a1f74925/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/cfea246829a33e932cd8a17c1650449bc3286fdaf81163e97eb072fbafbd3677/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/615522e265d04e67941244a2cb7d77b40141415dc77b4991c3c3b2b8b573cfe0/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/ff66a97ad0e23d6f5320707cb152a19046102b1f6e84321ab042a9f712ae71f2/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/d9f767edf48a3900d215fd9c6dc79733b6af06810fd621d265d63279f00165fa/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/239aaacf58c675242bd25ab6cd0e4428a80697fe0aa28a0b11560735a4c350c1/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/418cfc7dc84452ec58a63c9454eecb4ef223c9539d11257e0d33e9a89d6ad512/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/3a7e577deea3ebfae93d004d24ca11a3341b7a05fbef653c452a394becb523a2/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.217137    5078 manager.go:217] Machine: {Timestamp:2026-10-30 23:06:54.216297482 +0000 UTC m=+0.104203833 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099578 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23ca854740cb96c3d693895bf2d508 SystemUUID:ec23ca85-4740-cb96-c3d6-93895bf2d508 BootID:c2f5b0af-ff5e-4107-8a87-a1566c9b40f4 Filesystems:[{Device:/run/containers/storage/overlay-containers/6619f18877b5bbac0bcd2088d544fb1fcb3733f5c8c3432c8f1bdec4bb752e3a/userdata/shm DeviceMajor:0 DeviceMinor:85 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f1d8828fff8ad25304de019af726541636d3be49ad97ab157a702fee3798b60/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88a97cace2ee8aca50c00da9c49d90bcf2d4af5d848140b1cf2002340b83d516/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-333 DeviceMajor:0 DeviceMinor:333 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1918245f-b465-485f-9603-04ebe4846085/volumes/kubernetes.io~projected/kube-api-access-pmhb8 DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7711a49b12888c6f8c173ce50e08a979cc1d841d1d0218af21cc633134a29005/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a07ae84d-1e2a-4f1e-a763-e5363ccdd520/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:280 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a07ae84d-1e2a-4f1e-a763-e5363ccdd520/volumes/kubernetes.io~projected/kube-api-access-hzt62 DeviceMajor:0 DeviceMinor:285 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4e4bb115-b334-423e-86f1-1e1abb499b5d/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:295 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1636f9a77eb115dae1851b59480a76d79e9d59ee8225089d11ab34c04ecda05d/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b99dd331-1d6d-434c-9b02-1b7694955006/volumes/kubernetes.io~projected/kube-api-access-5bm7q DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4e4bb115-b334-423e-86f1-1e1abb499b5d/volumes/kubernetes.io~projected/kube-api-access-l2fzc DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-310 DeviceMajor:0 DeviceMinor:310 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91deced5-f22e-4148-98da-6a232be165c3/volumes/kubernetes.io~projected/kube-api-access-kqblk DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6d5aef87-7d40-4aa2-958a-1fb73d427e3b/volumes/kubernetes.io~projected/kube-api-access-pb2fp DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a07ae84d-1e2a-4f1e-a763-e5363ccdd520/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:284 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/58ec1adc-599b-4796-b12e-1cc85a834b15/volumes/kubernetes.io~projected/kube-api-access-mq95j DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/06742bba-beca-46c9-8614-2fa5973c4864/volumes/kubernetes.io~projected/kube-api-access-rqcxx DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a4e5e6789fe136c0f61393a6b0b46b93a51333e25d04454ec1d97192e25b68d4/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b99dd331-1d6d-434c-9b02-1b7694955006/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b07b7204cd39e9c41de4b968110698ec83ed0e02861ef87ee4c0d55d56a2fc13/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f3d048a7dac8692c140c08e6cdf2c6de069f73e17c8625044406e1ad267865a0/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d753418031592ca311fa0bc6cb8bcc039543c68e5ec696435c1dcbfc260c320/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/029540db69f9838505973c4170867cfbcf771e1eaf79070f10248300de5452c1/userdata/shm DeviceMajor:0 DeviceMinor:306 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e8d2abb44c70942aa77687775e2477d94c0dd840609a45ec23e29865c86c6d15/userdata/shm DeviceMajor:0 DeviceMinor:63 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cdee91d4-262d-4142-9340-4243df4586b6/volumes/kubernetes.io~projected/kube-api-access-ngj6h DeviceMajor:0 DeviceMinor:222 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2e942185-68e3-4d57-9f42-49d2965b41e4/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:304 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/108c7917-29b6-47bc-a590-35fd94c3de19/volumes/kubernetes.io~projected/kube-api-access-rqgxb DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70a51653-2167-4e11-bf04-81f5b10c9c9d/volumes/kubernetes.io~projected/kube-api-access-q46ww DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7e14c04d-b583-4f0a-840c-d493b2a2bbaf/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22aabe7f-74b4-4cc7-b88b-1ec78aecc925/volumes/kubernetes.io~projected/kube-api-access-z5p98 DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81b805c0-763e-4cd9-b6f5-af98767b298e/volumes/kubernetes.io~projected/kube-api-access-d6bjr DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3aba3250bfb20a60bc07e992dab181daa77b3a0019b1db9e6ad7c59a05535b39/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9bd99d0d-1f0f-4926-b835-880a64193ded/volumes/kubernetes.io~projected/kube-api-access-k2kbb DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2cd84d6c-d204-4e07-b591-9930cc9062a5/volumes/kubernetes.io~projected/kube-api-access-qhw29 DeviceMajor:0 DeviceMinor:276 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a07ae84d-1e2a-4f1e-a763-e5363ccdd520/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:282 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93660373ddb05521fe81f84cff70a8d616799ede9b3dd690af5a7617ad1befaf/userdata/shm DeviceMajor:0 DeviceMinor:300 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1633d8d790e534e1a6b63c58848a85702b22644518928f2c017cc79fe451df4d/userdata/shm DeviceMajor:0 DeviceMinor:308 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81b805c0-763e-4cd9-b6f5-af98767b298e/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/81b805c0-763e-4cd9-b6f5-af98767b298e/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/577bbe5f783c849e7733d86c2d6569bb0b9955764cc058b2a7c01a5fe3dda90c/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4f785b83-23af-45c1-975c-715bdebea140/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:254 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/995c387048a10c8ceea3500e3253ea660cf67ace643da0c2aded683099e26795/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/06742bba-beca-46c9-8614-2fa5973c4864/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:228 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a07ae84d-1e2a-4f1e-a763-e5363ccdd520/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:281 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b83d6f660917e9f5f7b17c578365082c77ecf1744fd9a8adf73375b0bcbd39f/userdata/shm DeviceMajor:0 DeviceMinor:283 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf/volumes/kubernetes.io~projected/kube-api-access-9mdsg DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/df39047e-a25e-4661-a34e-a6bca0f344f2/volumes/kubernetes.io~projected/kube-api-access-p5zjd DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9f28f4ffabe36c7eb7853ab824409f4c93fc666101154ea2a9de302865ebb19a/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-107 DeviceMajor:0 DeviceMinor:107 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22aabe7f-74b4-4cc7-b88b-1ec78aecc925/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c0ebf017ce4c7526dddc3e8f9d6f9518b84b7ca69a5085f1e40912bbfd5fc03/userdata/shm DeviceMajor:0 DeviceMinor:62 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7de3c861ceb745ee60fac5d3355fbaf7a32b92c2101ba5aa447c370dbc1efe12/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4f785b83-23af-45c1-975c-715bdebea140/volumes/kubernetes.io~projected/kube-api-access-nzppf DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/22aabe7f-74b4-4cc7-b88b-1ec78aecc925/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f88938488131bbe50049327147b95a87b12c1e36a3b0941d9062a772190f1524/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58ec1adc-599b-4796-b12e-1cc85a834b15/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:305 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:029540db69f9838 MacAddress:82:ef:18:41:49:6b Speed:10000 Mtu:8901} {Name:1633d8d790e534e MacAddress:ca:9e:fd:85:ea:26 Speed:10000 Mtu:8901} {Name:1636f9a77eb115d MacAddress:42:c9:b7:8f:0c:61 Speed:10000 Mtu:8901} {Name:3aba3250bfb20a6 MacAddress:a2:b8:23:bc:bb:92 Speed:10000 Mtu:8901} {Name:88a97cace2ee8ac MacAddress:62:d7:2e:cb:07:a1 Speed:10000 Mtu:8901} {Name:8b83d6f660917e9 MacAddress:16:3e:b0:e9:62:9c Speed:10000 Mtu:8901} {Name:93660373ddb0552 MacAddress:da:95:5d:2d:e3:4c Speed:10000 Mtu:8901} {Name:995c387048a10c8 MacAddress:32:71:9f:01:c6:b0 Speed:10000 Mtu:8901} {Name:9f1d8828fff8ad2 MacAddress:b2:39:cb:6b:65:4a Speed:10000 Mtu:8901} {Name:b07b7204cd39e9c MacAddress:4e:3a:e7:51:5b:8f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:02:0a:8a:99:bb Speed:0 Mtu:9001} {Name:br-int MacAddress:86:b3:b6:98:07:88 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:02:0a:8a:99:bb Speed:0 Mtu:9001} {Name:f3d048a7dac8692 MacAddress:c2:d4:6a:ce:32:0f Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:8e:c4:3e:a9:f0:f4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2e:12:2a:c0:19:3a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.217733    5078 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.217835    5078 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.218233    5078 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.218267    5078 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-205.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.219115    5078 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.219133    5078 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.219149    5078 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.219177    5078 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.219283    5078 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.219507    5078 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.219672    5078 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.219734    5078 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.219748    5078 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.220106    5078 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.220145    5078 kubelet.go:405] "Adding apiserver pod source"
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.220160    5078 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.221359    5078 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.221373    5078 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:06:54.223633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.222506    5078 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 23:06:54.233500 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.233480    5078 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 23:06:54.233500 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.233499    5078 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 23:06:54.233668 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.233646    5078 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 23:06:54.233895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.233881    5078 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 23:06:54.233941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.233913    5078 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 23:06:54.234031 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.234012    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 23:06:54.234031 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.234032    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 23:06:54.234178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.234043    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 23:06:54.234178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.234053    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 23:06:54.234178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.234061    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 23:06:54.234178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.234067    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 23:06:54.234178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.234073    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 23:06:54.234178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.234081    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 23:06:54.234178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.234092    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 23:06:54.234178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.234100    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 23:06:54.234178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.234114    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 23:06:54.234178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.234124    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 23:06:54.234178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.234147    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 23:06:54.234178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.234153    5078 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 23:06:54.238092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.238078    5078 server.go:1263] "Started kubelet"
Oct 30 23:06:54.238189 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.238159    5078 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 23:06:54.238495 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.238440    5078 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 23:06:54.238583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.238520    5078 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 23:06:54.239015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.238952    5078 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 23:06:54.239771 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.239754    5078 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 23:06:54.241144 ip-10-0-130-205 systemd[1]: Started Kubernetes Kubelet.
Oct 30 23:06:54.243870 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.241196    5078 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 23:06:54.243870 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:06:54.241992    5078 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-205.us-west-1.compute.internal\" not found"
Oct 30 23:06:54.243870 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.242037    5078 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 23:06:54.243870 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.242045    5078 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 23:06:54.243870 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.242422    5078 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.246240    5078 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.246705    5078 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:00:43 +0000 UTC" deadline="2028-05-01 12:24:44.426158497 +0000 UTC"
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.246740    5078 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13165h17m50.179422234s"
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249181    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bd99d0d-1f0f-4926-b835-880a64193ded" volumeName="kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249235    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a07ae84d-1e2a-4f1e-a763-e5363ccdd520" volumeName="kubernetes.io/projected/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-bound-sa-token" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249270    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06742bba-beca-46c9-8614-2fa5973c4864" volumeName="kubernetes.io/configmap/06742bba-beca-46c9-8614-2fa5973c4864-signing-cabundle" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249284    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22aabe7f-74b4-4cc7-b88b-1ec78aecc925" volumeName="kubernetes.io/configmap/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249345    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29261ce8-6b07-4ea5-afa4-8500cc1c7bbf" volumeName="kubernetes.io/configmap/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf-iptables-alerter-script" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249360    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e4bb115-b334-423e-86f1-1e1abb499b5d" volumeName="kubernetes.io/empty-dir/4e4bb115-b334-423e-86f1-1e1abb499b5d-tmp-dir" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249449    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f785b83-23af-45c1-975c-715bdebea140" volumeName="kubernetes.io/empty-dir/4f785b83-23af-45c1-975c-715bdebea140-data-volume" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249471    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58ec1adc-599b-4796-b12e-1cc85a834b15" volumeName="kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249508    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91deced5-f22e-4148-98da-6a232be165c3" volumeName="kubernetes.io/configmap/91deced5-f22e-4148-98da-6a232be165c3-multus-daemon-config" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249531    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df39047e-a25e-4661-a34e-a6bca0f344f2" volumeName="kubernetes.io/configmap/df39047e-a25e-4661-a34e-a6bca0f344f2-serviceca" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249578    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f785b83-23af-45c1-975c-715bdebea140" volumeName="kubernetes.io/configmap/4f785b83-23af-45c1-975c-715bdebea140-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249618    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f785b83-23af-45c1-975c-715bdebea140" volumeName="kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249636    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22aabe7f-74b4-4cc7-b88b-1ec78aecc925" volumeName="kubernetes.io/configmap/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-env-overrides" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249668    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e4bb115-b334-423e-86f1-1e1abb499b5d" volumeName="kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249706    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70a51653-2167-4e11-bf04-81f5b10c9c9d" volumeName="kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249877    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e14c04d-b583-4f0a-840c-d493b2a2bbaf" volumeName="kubernetes.io/configmap/7e14c04d-b583-4f0a-840c-d493b2a2bbaf-konnectivity-ca" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249917    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81b805c0-763e-4cd9-b6f5-af98767b298e" volumeName="kubernetes.io/empty-dir/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-tuned" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249932    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91deced5-f22e-4148-98da-6a232be165c3" volumeName="kubernetes.io/configmap/91deced5-f22e-4148-98da-6a232be165c3-cni-binary-copy" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249967    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4f785b83-23af-45c1-975c-715bdebea140" volumeName="kubernetes.io/projected/4f785b83-23af-45c1-975c-715bdebea140-kube-api-access-nzppf" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.249983    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d5aef87-7d40-4aa2-958a-1fb73d427e3b" volumeName="kubernetes.io/configmap/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.250015    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81b805c0-763e-4cd9-b6f5-af98767b298e" volumeName="kubernetes.io/empty-dir/81b805c0-763e-4cd9-b6f5-af98767b298e-tmp" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.250033    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a07ae84d-1e2a-4f1e-a763-e5363ccdd520" volumeName="kubernetes.io/configmap/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-trusted-ca" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.250085    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a07ae84d-1e2a-4f1e-a763-e5363ccdd520" volumeName="kubernetes.io/configmap/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-registry-certificates" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.250147    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a07ae84d-1e2a-4f1e-a763-e5363ccdd520" volumeName="kubernetes.io/secret/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-image-registry-private-configuration" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.250164    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06742bba-beca-46c9-8614-2fa5973c4864" volumeName="kubernetes.io/secret/06742bba-beca-46c9-8614-2fa5973c4864-signing-key" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251042    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="108c7917-29b6-47bc-a590-35fd94c3de19" volumeName="kubernetes.io/empty-dir/108c7917-29b6-47bc-a590-35fd94c3de19-tmp-dir" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251087    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cd84d6c-d204-4e07-b591-9930cc9062a5" volumeName="kubernetes.io/empty-dir/2cd84d6c-d204-4e07-b591-9930cc9062a5-tmp" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251120    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58ec1adc-599b-4796-b12e-1cc85a834b15" volumeName="kubernetes.io/projected/58ec1adc-599b-4796-b12e-1cc85a834b15-kube-api-access-mq95j" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251220    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d5aef87-7d40-4aa2-958a-1fb73d427e3b" volumeName="kubernetes.io/configmap/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251258    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d5aef87-7d40-4aa2-958a-1fb73d427e3b" volumeName="kubernetes.io/projected/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-kube-api-access-pb2fp" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251331    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91deced5-f22e-4148-98da-6a232be165c3" volumeName="kubernetes.io/projected/91deced5-f22e-4148-98da-6a232be165c3-kube-api-access-kqblk" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251361    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a07ae84d-1e2a-4f1e-a763-e5363ccdd520" volumeName="kubernetes.io/projected/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-registry-tls" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251383    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22aabe7f-74b4-4cc7-b88b-1ec78aecc925" volumeName="kubernetes.io/secret/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251400    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bd99d0d-1f0f-4926-b835-880a64193ded" volumeName="kubernetes.io/empty-dir/9bd99d0d-1f0f-4926-b835-880a64193ded-archives-path" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251421    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bd99d0d-1f0f-4926-b835-880a64193ded" volumeName="kubernetes.io/projected/9bd99d0d-1f0f-4926-b835-880a64193ded-kube-api-access-k2kbb" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251436    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a07ae84d-1e2a-4f1e-a763-e5363ccdd520" volumeName="kubernetes.io/empty-dir/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-ca-trust-extracted" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251458    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a07ae84d-1e2a-4f1e-a763-e5363ccdd520" volumeName="kubernetes.io/projected/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-kube-api-access-hzt62" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251505    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b99dd331-1d6d-434c-9b02-1b7694955006" volumeName="kubernetes.io/projected/b99dd331-1d6d-434c-9b02-1b7694955006-kube-api-access-5bm7q" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251583    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b99dd331-1d6d-434c-9b02-1b7694955006" volumeName="kubernetes.io/secret/b99dd331-1d6d-434c-9b02-1b7694955006-cert" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251625    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1918245f-b465-485f-9603-04ebe4846085" volumeName="kubernetes.io/projected/1918245f-b465-485f-9603-04ebe4846085-kube-api-access-pmhb8" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251640    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22aabe7f-74b4-4cc7-b88b-1ec78aecc925" volumeName="kubernetes.io/projected/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-kube-api-access-z5p98" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251658    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29261ce8-6b07-4ea5-afa4-8500cc1c7bbf" volumeName="kubernetes.io/projected/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf-kube-api-access-9mdsg" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251672    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2cd84d6c-d204-4e07-b591-9930cc9062a5" volumeName="kubernetes.io/projected/2cd84d6c-d204-4e07-b591-9930cc9062a5-kube-api-access-qhw29" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251771    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e942185-68e3-4d57-9f42-49d2965b41e4" volumeName="kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251791    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e4bb115-b334-423e-86f1-1e1abb499b5d" volumeName="kubernetes.io/projected/4e4bb115-b334-423e-86f1-1e1abb499b5d-kube-api-access-l2fzc" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251808    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7e14c04d-b583-4f0a-840c-d493b2a2bbaf" volumeName="kubernetes.io/secret/7e14c04d-b583-4f0a-840c-d493b2a2bbaf-agent-certs" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251827    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81b805c0-763e-4cd9-b6f5-af98767b298e" volumeName="kubernetes.io/projected/81b805c0-763e-4cd9-b6f5-af98767b298e-kube-api-access-d6bjr" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251842    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06742bba-beca-46c9-8614-2fa5973c4864" volumeName="kubernetes.io/projected/06742bba-beca-46c9-8614-2fa5973c4864-kube-api-access-rqcxx" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251857    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="108c7917-29b6-47bc-a590-35fd94c3de19" volumeName="kubernetes.io/projected/108c7917-29b6-47bc-a590-35fd94c3de19-kube-api-access-rqgxb" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251875    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e4bb115-b334-423e-86f1-1e1abb499b5d" volumeName="kubernetes.io/configmap/4e4bb115-b334-423e-86f1-1e1abb499b5d-config-volume" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251892    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bd99d0d-1f0f-4926-b835-880a64193ded" volumeName="kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-service-ca-bundle" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251916    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a07ae84d-1e2a-4f1e-a763-e5363ccdd520" volumeName="kubernetes.io/secret/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-installation-pull-secrets" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251932    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cdee91d4-262d-4142-9340-4243df4586b6" volumeName="kubernetes.io/projected/cdee91d4-262d-4142-9340-4243df4586b6-kube-api-access-ngj6h" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251946    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df39047e-a25e-4661-a34e-a6bca0f344f2" volumeName="kubernetes.io/projected/df39047e-a25e-4661-a34e-a6bca0f344f2-kube-api-access-p5zjd" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251966    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22aabe7f-74b4-4cc7-b88b-1ec78aecc925" volumeName="kubernetes.io/configmap/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-ovnkube-config" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251983    5078 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d5aef87-7d40-4aa2-958a-1fb73d427e3b" volumeName="kubernetes.io/configmap/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-cni-binary-copy" seLinuxMountContext=""
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.251995    5078 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 23:06:54.252098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.252005    5078 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 23:06:54.254909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.252862    5078 factory.go:55] Registering systemd factory
Oct 30 23:06:54.254909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.252883    5078 factory.go:223] Registration of the systemd container factory successfully
Oct 30 23:06:54.255079 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.255004    5078 factory.go:153] Registering CRI-O factory
Oct 30 23:06:54.255079 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.255026    5078 factory.go:223] Registration of the crio container factory successfully
Oct 30 23:06:54.255079 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.255070    5078 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 23:06:54.256245 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.255088    5078 factory.go:103] Registering Raw factory
Oct 30 23:06:54.256245 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.255100    5078 manager.go:1199] Started watching for new ooms in manager
Oct 30 23:06:54.256245 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.255397    5078 manager.go:319] Starting recovery of all containers
Oct 30 23:06:54.284626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.280464    5078 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 23:06:54.290862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.290358    5078 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 23:06:54.290862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.290387    5078 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 23:06:54.290862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.290411    5078 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 23:06:54.290862 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:06:54.290451    5078 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 23:06:54.313883 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.313803    5078 generic.go:356] "Generic (PLEG): container finished" podID="6d5aef87-7d40-4aa2-958a-1fb73d427e3b" containerID="83701692091b14f4ea64ac22ecf562beede8e17c5efc134cc2b4081329d48060" exitCode=0
Oct 30 23:06:54.313883 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.313839    5078 generic.go:356] "Generic (PLEG): container finished" podID="6d5aef87-7d40-4aa2-958a-1fb73d427e3b" containerID="4aff3584c5c385b309542300b7adecf952f644dd455110ecfb2d954875e27805" exitCode=0
Oct 30 23:06:54.313883 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.313850    5078 generic.go:356] "Generic (PLEG): container finished" podID="6d5aef87-7d40-4aa2-958a-1fb73d427e3b" containerID="46d7dd8e9987eb646077f0f9458e152722cfd4f4331d250672e87bf084c6c5dc" exitCode=0
Oct 30 23:06:54.313883 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.313856    5078 generic.go:356] "Generic (PLEG): container finished" podID="6d5aef87-7d40-4aa2-958a-1fb73d427e3b" containerID="2b944907ecda24ca4f78216a09a75e77902a3631bf71f84789f0ed571e22e3c7" exitCode=0
Oct 30 23:06:54.313883 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.313864    5078 generic.go:356] "Generic (PLEG): container finished" podID="6d5aef87-7d40-4aa2-958a-1fb73d427e3b" containerID="31dd7921dfbc997e0fd40c328c1095805daea044b6d5758886238c28d6d9b797" exitCode=0
Oct 30 23:06:54.313883 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.313869    5078 generic.go:356] "Generic (PLEG): container finished" podID="6d5aef87-7d40-4aa2-958a-1fb73d427e3b" containerID="907cbd329009711b667f1ef4fd482bf3893f4af473e99633952dfaea0d7cbe5c" exitCode=0
Oct 30 23:06:54.317743 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.317710    5078 generic.go:356] "Generic (PLEG): container finished" podID="8bc6014254655639fce61607de5fe088" containerID="2c05da7699bbb3c4abc65436e7b2d424b7e38acd1d39300ddc3bc6f0c3029a73" exitCode=0
Oct 30 23:06:54.342109 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:06:54.342079    5078 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-205.us-west-1.compute.internal\" not found"
Oct 30 23:06:54.353908 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.353887    5078 manager.go:324] Recovery completed
Oct 30 23:06:54.374173 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.374148    5078 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:06:54.376145 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.376119    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:06:54.376249 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.376157    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:06:54.376249 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.376172    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:06:54.378663 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.378645    5078 cpu_manager.go:226] "Starting" policy="none"
Oct 30 23:06:54.378663 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.378658    5078 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 23:06:54.378663 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.378660    5078 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:06:54.378877 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.378674    5078 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 23:06:54.378877 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.378830    5078 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 23:06:54.378877 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.378845    5078 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 23:06:54.378877 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.378863    5078 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 23:06:54.378877 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.378869    5078 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 23:06:54.378877 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.378876    5078 policy_none.go:50] "Start"
Oct 30 23:06:54.379083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.378893    5078 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 23:06:54.379083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.378903    5078 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 23:06:54.379083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.379047    5078 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 23:06:54.379083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.379055    5078 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 23:06:54.379083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.379060    5078 policy_none.go:44] "Start"
Oct 30 23:06:54.380909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.380891    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:06:54.381000 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.380923    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:06:54.381000 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.380943    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:06:54.384515 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.384481    5078 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 23:06:54.387305 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.387291    5078 manager.go:341] "Starting Device Plugin manager"
Oct 30 23:06:54.387421 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:06:54.387321    5078 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 23:06:54.387421 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.387332    5078 server.go:87] "Starting device plugin registration server"
Oct 30 23:06:54.387696 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.387682    5078 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 23:06:54.387785 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.387696    5078 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 23:06:54.387902 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.387889    5078 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 23:06:54.388007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.387979    5078 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 23:06:54.388069 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.388020    5078 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 23:06:54.388069 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.388028    5078 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 23:06:54.388166 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.388061    5078 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:06:54.388005823Z","UUID":"1207d2df-3a72-4b31-a469-52c1ae0fc2ec","Handler":null,"Name":"","Endpoint":""}
Oct 30 23:06:54.389289 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.389274    5078 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 23:06:54.389360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.389295    5078 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 23:06:54.389524 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:06:54.389470    5078 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 23:06:54.389524 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:06:54.389507    5078 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-205.us-west-1.compute.internal\" not found"
Oct 30 23:06:54.391149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.390571    5078 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal"]
Oct 30 23:06:54.391341 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.391302    5078 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:06:54.393169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.393149    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:06:54.393278 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.393260    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:06:54.393377 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.393286    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:06:54.394115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.394097    5078 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:06:54.394288 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.394274    5078 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:06:54.395361 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.395340    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:06:54.395458 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.395384    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:06:54.395458 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.395400    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:06:54.395706 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.395687    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:06:54.395766 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.395715    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:06:54.395766 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.395729    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:06:54.395766 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.395717    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal" event={"ID":"8bc6014254655639fce61607de5fe088","Type":"ContainerStarted","Data":"83ca72038a6ecbeb4c2cf5d1e0d543ef3744f4bce8e946dba4d340a2f583ff38"}
Oct 30 23:06:54.395766 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.395754    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal" event={"ID":"8bc6014254655639fce61607de5fe088","Type":"ContainerDied","Data":"2c05da7699bbb3c4abc65436e7b2d424b7e38acd1d39300ddc3bc6f0c3029a73"}
Oct 30 23:06:54.396025 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.395771    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal" event={"ID":"8bc6014254655639fce61607de5fe088","Type":"ContainerStarted","Data":"0c0ebf017ce4c7526dddc3e8f9d6f9518b84b7ca69a5085f1e40912bbfd5fc03"}
Oct 30 23:06:54.396025 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.395783    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal" event={"ID":"a1b66be07f6b0e09e908458ab3790deb","Type":"ContainerStarted","Data":"9e5235d72e5e721ca45d2fff3566007931ae9bc547833d1498aa2c2ebb7f8c18"}
Oct 30 23:06:54.396025 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.395794    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal" event={"ID":"a1b66be07f6b0e09e908458ab3790deb","Type":"ContainerStarted","Data":"9f28f4ffabe36c7eb7853ab824409f4c93fc666101154ea2a9de302865ebb19a"}
Oct 30 23:06:54.396025 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.395816    5078 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:06:54.396025 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:06:54.395881    5078 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-205.us-west-1.compute.internal\" not found" node="ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:54.396580 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.396559    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:06:54.396758 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.396613    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:06:54.396758 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.396630    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:06:54.396758 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:06:54.396751    5078 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-205.us-west-1.compute.internal\" not found" node="ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:54.431301 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.431273    5078 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:06:54.431438 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.431362    5078 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:06:54.446745 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.446721    5078 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:54.485375 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.485271    5078 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:06:54.485817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.485790    5078 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 23:06:54.485951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.485941    5078 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 23:06:54.487956 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.487938    5078 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 23:06:54.489029 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.489006    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 30 23:06:54.489129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.489041    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 23:06:54.489129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.489056    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 30 23:06:54.489129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.489107    5078 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:54.530818 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.530793    5078 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:06:54.530990 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:06:54.530853    5078 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:54.530990 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.530870    5078 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:54.586639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.586482    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bc6014254655639fce61607de5fe088-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal\" (UID: \"8bc6014254655639fce61607de5fe088\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:54.586639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.586533    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bc6014254655639fce61607de5fe088-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal\" (UID: \"8bc6014254655639fce61607de5fe088\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:54.586639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.586558    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1b66be07f6b0e09e908458ab3790deb-config\") pod \"kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal\" (UID: \"a1b66be07f6b0e09e908458ab3790deb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:54.604882 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.604849    5078 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:54.605023 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.604931    5078 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:54.605586 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.605561    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeNotReady"
Oct 30 23:06:54.605702 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.605611    5078 setters.go:546] "Node became not ready" node="ip-10-0-130-205.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T23:06:54Z","lastTransitionTime":"2026-10-30T23:06:54Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 23:06:54.608498 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.608469    5078 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:06:54.608625 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:06:54.608525    5078 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:54.646629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.646576    5078 nodeinfomanager.go:471] existing CSINode "ip-10-0-130-205.us-west-1.compute.internal" is owned by different node (oldNodeID="63cf1c1b-40b6-4e83-8770-bdb8555e7f91", newNodeID=""), cleaning up...
Oct 30 23:06:54.687755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.687723    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bc6014254655639fce61607de5fe088-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal\" (UID: \"8bc6014254655639fce61607de5fe088\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:54.687755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.687743    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8bc6014254655639fce61607de5fe088-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal\" (UID: \"8bc6014254655639fce61607de5fe088\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:54.687973 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.687805    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bc6014254655639fce61607de5fe088-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal\" (UID: \"8bc6014254655639fce61607de5fe088\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:54.687973 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.687831    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1b66be07f6b0e09e908458ab3790deb-config\") pod \"kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal\" (UID: \"a1b66be07f6b0e09e908458ab3790deb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:54.687973 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.687872    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a1b66be07f6b0e09e908458ab3790deb-config\") pod \"kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal\" (UID: \"a1b66be07f6b0e09e908458ab3790deb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:54.687973 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:54.687906    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bc6014254655639fce61607de5fe088-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal\" (UID: \"8bc6014254655639fce61607de5fe088\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:55.221488 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.221454    5078 apiserver.go:52] "Watching apiserver"
Oct 30 23:06:55.366785 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.366756    5078 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:55.366946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.366926    5078 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:55.368511 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.368488    5078 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 23:06:55.369037 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.369015    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-m896h","openshift-ovn-kubernetes/ovnkube-node-4c5rm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79","openshift-dns/node-resolver-gldcx","openshift-insights/insights-runtime-extractor-tcf86","openshift-network-operator/iptables-alerter-dzklv","openshift-insights/periodic-gathering-7v4kl-bjfqg","openshift-image-registry/node-ca-2g2gr","kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal","openshift-multus/network-metrics-daemon-pln6h","openshift-console/downloads-6dd789c8b7-9vzv5","openshift-multus/multus-additional-cni-plugins-z7d8k","openshift-cluster-node-tuning-operator/tuned-rcgrg","openshift-multus/multus-wk66h","openshift-network-diagnostics/network-check-target-qvbgk","kube-system/global-pull-secret-syncer-cthfm","openshift-service-ca/service-ca-5fffc97bdf-9jlqk","openshift-ingress-canary/ingress-canary-xslb9","openshift-monitoring/node-exporter-6bv5g","openshift-dns/dns-default-8qhk5","openshift-image-registry/image-registry-7f5789fcdd-txvlr","kube-system/konnectivity-agent-fg8fg"]
Oct 30 23:06:55.383714 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.383688    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.424756 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.424728    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 23:06:55.427815 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.427794    5078 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:06:55.427941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.427821    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 23:06:55.427941 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:06:55.427845    5078 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:55.428089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.428068    5078 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 23:06:55.428212 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:06:55.428107    5078 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-205.us-west-1.compute.internal"
Oct 30 23:06:55.428447 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.428426    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 23:06:55.428949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.428859    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 23:06:55.428949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.428871    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 23:06:55.428949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.428902    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 23:06:55.428949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.428922    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 23:06:55.428949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.428930    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 23:06:55.429225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.428976    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 23:06:55.429225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.428982    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 23:06:55.429225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.429022    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 23:06:55.429225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.429034    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 23:06:55.429225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.429096    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 23:06:55.429225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.429096    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 23:06:55.429225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.429145    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 23:06:55.429225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.429150    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 23:06:55.429225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.429190    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:06:55.429583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.429237    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 23:06:55.429583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.429249    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 23:06:55.429583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.429262    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 23:06:55.429583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.429298    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 23:06:55.429583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.429299    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 23:06:55.429583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.429411    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 23:06:55.445729 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.445706    5078 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 23:06:55.446109 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.446088    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-trr47\""
Oct 30 23:06:55.446185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.446129    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fbzzd\""
Oct 30 23:06:55.446185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.446135    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tw7s4\""
Oct 30 23:06:55.446185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.446181    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7m4ss\""
Oct 30 23:06:55.446312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.446203    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 23:06:55.446312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.446225    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ldcvv\""
Oct 30 23:06:55.446312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.446268    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7sr97\""
Oct 30 23:06:55.446312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.446278    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-46zpf\""
Oct 30 23:06:55.446312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.446301    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-4qgbv\""
Oct 30 23:06:55.446504 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.446331    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-t7twr\""
Oct 30 23:06:55.446504 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.446382    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 23:06:55.446504 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.446393    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 23:06:55.446504 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.446395    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4j2qm\""
Oct 30 23:06:55.469217 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.469188    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 23:06:55.469371 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.469356    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cqcdm\""
Oct 30 23:06:55.469437 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.469420    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 23:06:55.469492 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.469459    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6m9zb\""
Oct 30 23:06:55.471164 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.471144    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 23:06:55.471253 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.471206    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-l6bwx\""
Oct 30 23:06:55.471253 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.471229    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-w9j6t\""
Oct 30 23:06:55.471253 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.471244    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 23:06:55.472103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.471997    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-cl4w9\""
Oct 30 23:06:55.472103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.472012    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-nlg56\""
Oct 30 23:06:55.472103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.472035    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-v482q\""
Oct 30 23:06:55.472103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.472002    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 23:06:55.472103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.472071    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 23:06:55.472392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.472158    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 23:06:55.472392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.472226    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 23:06:55.474272 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.474254    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-c9hs4\""
Oct 30 23:06:55.474374 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.474291    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s47dt\""
Oct 30 23:06:55.474374 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.474304    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9xqsx\""
Oct 30 23:06:55.474374 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.474329    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-7sq9v\""
Oct 30 23:06:55.474374 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.474313    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 23:06:55.491837 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.491804    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-sysctl-conf\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.491938 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.491837    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-modprobe-d\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.491938 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.491863    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqgxb\" (UniqueName: \"kubernetes.io/projected/108c7917-29b6-47bc-a590-35fd94c3de19-kube-api-access-rqgxb\") pod \"node-resolver-gldcx\" (UID: \"108c7917-29b6-47bc-a590-35fd94c3de19\") " pod="openshift-dns/node-resolver-gldcx"
Oct 30 23:06:55.491938 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.491880    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-ovn-node-metrics-cert\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.491938 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.491901    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/06742bba-beca-46c9-8614-2fa5973c4864-signing-key\") pod \"service-ca-5fffc97bdf-9jlqk\" (UID: \"06742bba-beca-46c9-8614-2fa5973c4864\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk"
Oct 30 23:06:55.492110 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.491956    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzt62\" (UniqueName: \"kubernetes.io/projected/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-kube-api-access-hzt62\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:55.492110 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492028    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f785b83-23af-45c1-975c-715bdebea140-crio-socket\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:55.492110 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492080    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf-host-slash\") pod \"iptables-alerter-dzklv\" (UID: \"29261ce8-6b07-4ea5-afa4-8500cc1c7bbf\") " pod="openshift-network-operator/iptables-alerter-dzklv"
Oct 30 23:06:55.492110 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492105    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91deced5-f22e-4148-98da-6a232be165c3-cni-binary-copy\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.492298 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492129    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:55.492298 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492149    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-run-ovn-kubernetes\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.492298 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492169    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-sysctl-d\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.492298 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492195    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7e14c04d-b583-4f0a-840c-d493b2a2bbaf-agent-certs\") pod \"konnectivity-agent-fg8fg\" (UID: \"7e14c04d-b583-4f0a-840c-d493b2a2bbaf\") " pod="kube-system/konnectivity-agent-fg8fg"
Oct 30 23:06:55.492298 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492220    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-kubernetes\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.492298 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492238    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-etc-selinux\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.492298 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492253    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-run-netns\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.492298 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492268    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e942185-68e3-4d57-9f42-49d2965b41e4-dbus\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:55.492298 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492284    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-var-lib-openvswitch\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.492742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492308    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzppf\" (UniqueName: \"kubernetes.io/projected/4f785b83-23af-45c1-975c-715bdebea140-kube-api-access-nzppf\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:55.492742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492322    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-registration-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.492742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492338    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-cnibin\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:06:55.492742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492363    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-node-exporter-accelerators-collector-config\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.492742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492417    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-ovn-node-metrics-cert\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.492742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492489    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/91deced5-f22e-4148-98da-6a232be165c3-cni-binary-copy\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.492742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492572    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/06742bba-beca-46c9-8614-2fa5973c4864-signing-key\") pod \"service-ca-5fffc97bdf-9jlqk\" (UID: \"06742bba-beca-46c9-8614-2fa5973c4864\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk"
Oct 30 23:06:55.492742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492628    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.492742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492646    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7e14c04d-b583-4f0a-840c-d493b2a2bbaf-agent-certs\") pod \"konnectivity-agent-fg8fg\" (UID: \"7e14c04d-b583-4f0a-840c-d493b2a2bbaf\") " pod="kube-system/konnectivity-agent-fg8fg"
Oct 30 23:06:55.492742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492678    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-registry-certificates\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:55.492742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492711    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-service-ca-bundle\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:55.492742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492730    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e942185-68e3-4d57-9f42-49d2965b41e4-original-pull-secret\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:55.493161 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492762    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:55.493161 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492804    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-systemd\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.493161 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492919    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cd84d6c-d204-4e07-b591-9930cc9062a5-tmp\") pod \"downloads-6dd789c8b7-9vzv5\" (UID: \"2cd84d6c-d204-4e07-b591-9930cc9062a5\") " pod="openshift-console/downloads-6dd789c8b7-9vzv5"
Oct 30 23:06:55.493161 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492980    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs\") pod \"network-metrics-daemon-pln6h\" (UID: \"58ec1adc-599b-4796-b12e-1cc85a834b15\") " pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:06:55.493161 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.492995    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4f785b83-23af-45c1-975c-715bdebea140-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:55.493161 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493025    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2cd84d6c-d204-4e07-b591-9930cc9062a5-tmp\") pod \"downloads-6dd789c8b7-9vzv5\" (UID: \"2cd84d6c-d204-4e07-b591-9930cc9062a5\") " pod="openshift-console/downloads-6dd789c8b7-9vzv5"
Oct 30 23:06:55.493161 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493082    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91deced5-f22e-4148-98da-6a232be165c3-multus-daemon-config\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.493383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493171    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4bb115-b334-423e-86f1-1e1abb499b5d-config-volume\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:55.493383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493210    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81b805c0-763e-4cd9-b6f5-af98767b298e-tmp\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.493383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493232    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f785b83-23af-45c1-975c-715bdebea140-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:55.493383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493242    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58ec1adc-599b-4796-b12e-1cc85a834b15-metrics-certs\") pod \"network-metrics-daemon-pln6h\" (UID: \"58ec1adc-599b-4796-b12e-1cc85a834b15\") " pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:06:55.493383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493293    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-multus-socket-dir-parent\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.493383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493296    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/81b805c0-763e-4cd9-b6f5-af98767b298e-tmp\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.493383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493309    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/91deced5-f22e-4148-98da-6a232be165c3-multus-daemon-config\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.493383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493326    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-trusted-ca-bundle\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:55.493383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493363    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:06:55.493383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493385    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-installation-pull-secrets\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:55.493858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493401    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b99dd331-1d6d-434c-9b02-1b7694955006-cert\") pod \"ingress-canary-xslb9\" (UID: \"b99dd331-1d6d-434c-9b02-1b7694955006\") " pod="openshift-ingress-canary/ingress-canary-xslb9"
Oct 30 23:06:55.493858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493425    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f785b83-23af-45c1-975c-715bdebea140-data-volume\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:55.493858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493483    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4f785b83-23af-45c1-975c-715bdebea140-data-volume\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:55.493858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493523    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:06:55.493858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493563    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-sys-fs\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.493858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493621    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqblk\" (UniqueName: \"kubernetes.io/projected/91deced5-f22e-4148-98da-6a232be165c3-kube-api-access-kqblk\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.493858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493654    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-log-socket\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.493858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493672    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-installation-pull-secrets\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:55.493858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493690    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b99dd331-1d6d-434c-9b02-1b7694955006-cert\") pod \"ingress-canary-xslb9\" (UID: \"b99dd331-1d6d-434c-9b02-1b7694955006\") " pod="openshift-ingress-canary/ingress-canary-xslb9"
Oct 30 23:06:55.493858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493681    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9bd99d0d-1f0f-4926-b835-880a64193ded-archives-path\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:55.493858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493737    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-sys\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.493858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493757    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9bd99d0d-1f0f-4926-b835-880a64193ded-archives-path\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:55.493858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493768    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bm7q\" (UniqueName: \"kubernetes.io/projected/b99dd331-1d6d-434c-9b02-1b7694955006-kube-api-access-5bm7q\") pod \"ingress-canary-xslb9\" (UID: \"b99dd331-1d6d-434c-9b02-1b7694955006\") " pod="openshift-ingress-canary/ingress-canary-xslb9"
Oct 30 23:06:55.493858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493793    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb2fp\" (UniqueName: \"kubernetes.io/projected/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-kube-api-access-pb2fp\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:06:55.493858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493817    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-cni-bin\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.493858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493846    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/06742bba-beca-46c9-8614-2fa5973c4864-signing-cabundle\") pod \"service-ca-5fffc97bdf-9jlqk\" (UID: \"06742bba-beca-46c9-8614-2fa5973c4864\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk"
Oct 30 23:06:55.494456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493900    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-root\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.494456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493915    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-metrics-client-ca\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.494456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493937    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-run\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.494456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.493994    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6bjr\" (UniqueName: \"kubernetes.io/projected/81b805c0-763e-4cd9-b6f5-af98767b298e-kube-api-access-d6bjr\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.494456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494125    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-run-k8s-cni-cncf-io\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.494456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494189    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-node-log\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.494456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494225    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-node-exporter-tls\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.494456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494254    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:06:55.494456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494284    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-bound-sa-token\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:55.494456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494300    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhw29\" (UniqueName: \"kubernetes.io/projected/2cd84d6c-d204-4e07-b591-9930cc9062a5-kube-api-access-qhw29\") pod \"downloads-6dd789c8b7-9vzv5\" (UID: \"2cd84d6c-d204-4e07-b591-9930cc9062a5\") " pod="openshift-console/downloads-6dd789c8b7-9vzv5"
Oct 30 23:06:55.494456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494317    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/108c7917-29b6-47bc-a590-35fd94c3de19-tmp-dir\") pod \"node-resolver-gldcx\" (UID: \"108c7917-29b6-47bc-a590-35fd94c3de19\") " pod="openshift-dns/node-resolver-gldcx"
Oct 30 23:06:55.494456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494332    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q46ww\" (UniqueName: \"kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww\") pod \"network-check-target-qvbgk\" (UID: \"70a51653-2167-4e11-bf04-81f5b10c9c9d\") " pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:06:55.494456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494347    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-ovnkube-script-lib\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.494456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494427    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/108c7917-29b6-47bc-a590-35fd94c3de19-tmp-dir\") pod \"node-resolver-gldcx\" (UID: \"108c7917-29b6-47bc-a590-35fd94c3de19\") " pod="openshift-dns/node-resolver-gldcx"
Oct 30 23:06:55.494456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494450    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-node-exporter-textfile\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.495101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494484    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-socket-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.495101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494512    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-kubelet\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.495101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494525    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-cni-binary-copy\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:06:55.495101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494553    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7e14c04d-b583-4f0a-840c-d493b2a2bbaf-konnectivity-ca\") pod \"konnectivity-agent-fg8fg\" (UID: \"7e14c04d-b583-4f0a-840c-d493b2a2bbaf\") " pod="kube-system/konnectivity-agent-fg8fg"
Oct 30 23:06:55.495101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494574    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-ovnkube-script-lib\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.495101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494583    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-multus-conf-dir\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.495101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494624    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-sys\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.495101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494721    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-run-systemd\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.495101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494783    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7e14c04d-b583-4f0a-840c-d493b2a2bbaf-konnectivity-ca\") pod \"konnectivity-agent-fg8fg\" (UID: \"7e14c04d-b583-4f0a-840c-d493b2a2bbaf\") " pod="kube-system/konnectivity-agent-fg8fg"
Oct 30 23:06:55.495101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494786    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-run-openvswitch\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.495101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494821    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5p98\" (UniqueName: \"kubernetes.io/projected/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-kube-api-access-z5p98\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.495101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494893    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-registry-tls\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:55.495101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.494980    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2kbb\" (UniqueName: \"kubernetes.io/projected/9bd99d0d-1f0f-4926-b835-880a64193ded-kube-api-access-k2kbb\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:55.495101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495016    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-run-ovn\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.495101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495041    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.495101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495062    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-registry-tls\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:55.495101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495067    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf-iptables-alerter-script\") pod \"iptables-alerter-dzklv\" (UID: \"29261ce8-6b07-4ea5-afa4-8500cc1c7bbf\") " pod="openshift-network-operator/iptables-alerter-dzklv"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495123    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495203    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-system-cni-dir\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495225    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-run-multus-certs\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495278    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495303    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-host\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495313    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495318    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf-iptables-alerter-script\") pod \"iptables-alerter-dzklv\" (UID: \"29261ce8-6b07-4ea5-afa4-8500cc1c7bbf\") " pod="openshift-network-operator/iptables-alerter-dzklv"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495348    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mdsg\" (UniqueName: \"kubernetes.io/projected/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf-kube-api-access-9mdsg\") pod \"iptables-alerter-dzklv\" (UID: \"29261ce8-6b07-4ea5-afa4-8500cc1c7bbf\") " pod="openshift-network-operator/iptables-alerter-dzklv"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495383    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq95j\" (UniqueName: \"kubernetes.io/projected/58ec1adc-599b-4796-b12e-1cc85a834b15-kube-api-access-mq95j\") pod \"network-metrics-daemon-pln6h\" (UID: \"58ec1adc-599b-4796-b12e-1cc85a834b15\") " pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495411    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/108c7917-29b6-47bc-a590-35fd94c3de19-hosts-file\") pod \"node-resolver-gldcx\" (UID: \"108c7917-29b6-47bc-a590-35fd94c3de19\") " pod="openshift-dns/node-resolver-gldcx"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495436    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495460    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-cnibin\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495482    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-var-lib-cni-multus\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495505    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-cni-netd\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495519    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4e4bb115-b334-423e-86f1-1e1abb499b5d-metrics-tls\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495550    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-trusted-ca\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495647    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-node-exporter-wtmp\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495704    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-var-lib-kubelet\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495752    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495784    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-lib-modules\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495837    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmhb8\" (UniqueName: \"kubernetes.io/projected/1918245f-b465-485f-9603-04ebe4846085-kube-api-access-pmhb8\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.495881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495875    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-multus-cni-dir\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.496755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.495989    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-etc-openvswitch\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.496755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496039    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqcxx\" (UniqueName: \"kubernetes.io/projected/06742bba-beca-46c9-8614-2fa5973c4864-kube-api-access-rqcxx\") pod \"service-ca-5fffc97bdf-9jlqk\" (UID: \"06742bba-beca-46c9-8614-2fa5973c4864\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk"
Oct 30 23:06:55.496755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496071    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-image-registry-private-configuration\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:55.496755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496098    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-systemd-units\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.496755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496122    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-sysconfig\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.496755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496146    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngj6h\" (UniqueName: \"kubernetes.io/projected/cdee91d4-262d-4142-9340-4243df4586b6-kube-api-access-ngj6h\") pod \"migrator-66df5d496b-m896h\" (UID: \"cdee91d4-262d-4142-9340-4243df4586b6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m896h"
Oct 30 23:06:55.496755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496186    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-env-overrides\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.496755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496236    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2fzc\" (UniqueName: \"kubernetes.io/projected/4e4bb115-b334-423e-86f1-1e1abb499b5d-kube-api-access-l2fzc\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:55.496755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496278    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-image-registry-private-configuration\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:55.496755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496367    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-env-overrides\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.496755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496382    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df39047e-a25e-4661-a34e-a6bca0f344f2-serviceca\") pod \"node-ca-2g2gr\" (UID: \"df39047e-a25e-4661-a34e-a6bca0f344f2\") " pod="openshift-image-registry/node-ca-2g2gr"
Oct 30 23:06:55.496755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496494    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5zjd\" (UniqueName: \"kubernetes.io/projected/df39047e-a25e-4661-a34e-a6bca0f344f2-kube-api-access-p5zjd\") pod \"node-ca-2g2gr\" (UID: \"df39047e-a25e-4661-a34e-a6bca0f344f2\") " pod="openshift-image-registry/node-ca-2g2gr"
Oct 30 23:06:55.496755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496538    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-device-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.496755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496562    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-hostroot\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.496755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496587    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e942185-68e3-4d57-9f42-49d2965b41e4-kubelet-config\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:55.496755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496749    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xcsm\" (UniqueName: \"kubernetes.io/projected/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-kube-api-access-2xcsm\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.497282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496796    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-tuned\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.497282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496833    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-system-cni-dir\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:06:55.497282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496883    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-tuned\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.497282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496892    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-slash\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.497282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.496962    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-run-netns\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.497282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.497233    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-ovnkube-config\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.497485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.497403    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-ovnkube-config\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.497485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.497439    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e4bb115-b334-423e-86f1-1e1abb499b5d-tmp-dir\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:55.497485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.497460    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-ca-trust-extracted\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:55.497485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.497476    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df39047e-a25e-4661-a34e-a6bca0f344f2-host\") pod \"node-ca-2g2gr\" (UID: \"df39047e-a25e-4661-a34e-a6bca0f344f2\") " pod="openshift-image-registry/node-ca-2g2gr"
Oct 30 23:06:55.497685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.497499    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-var-lib-cni-bin\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.497685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.497517    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-var-lib-kubelet\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.497685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.497538    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-etc-kubernetes\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.497685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.497590    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-ca-trust-extracted\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:55.497685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.497634    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e4bb115-b334-423e-86f1-1e1abb499b5d-tmp-dir\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:55.529441 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.529411    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 23:06:55.529633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.529472    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 23:06:55.530427 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.529701    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 23:06:55.530427 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.530376    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 23:06:55.530581 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.530479    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:06:55.530581 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.530523    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 23:06:55.531213 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.531134    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 23:06:55.532887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.531639    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 23:06:55.532887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.531873    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 23:06:55.532887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.532006    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 23:06:55.532887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.532053    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 23:06:55.532887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.532092    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 23:06:55.532887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.532273    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 23:06:55.532887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.532319    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 23:06:55.532887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.532415    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 23:06:55.532887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.532471    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 23:06:55.532887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.532525    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 23:06:55.532887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.532536    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 23:06:55.532887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.532685    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 23:06:55.532887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.532730    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 23:06:55.532887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.532794    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 23:06:55.532887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.532787    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 23:06:55.533553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.533228    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 23:06:55.533553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.533469    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-registry-certificates\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:55.533553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.533543    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-service-ca-bundle\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:55.533699 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.533621    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 23:06:55.533699 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.533645    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 23:06:55.534250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.533820    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4e4bb115-b334-423e-86f1-1e1abb499b5d-config-volume\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:55.534250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.533898    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4f785b83-23af-45c1-975c-715bdebea140-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:55.534250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.534059    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/06742bba-beca-46c9-8614-2fa5973c4864-signing-cabundle\") pod \"service-ca-5fffc97bdf-9jlqk\" (UID: \"06742bba-beca-46c9-8614-2fa5973c4864\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk"
Oct 30 23:06:55.534398 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.534334    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-trusted-ca-bundle\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:55.536684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.536665    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-trusted-ca\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:55.537226 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.537205    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df39047e-a25e-4661-a34e-a6bca0f344f2-serviceca\") pod \"node-ca-2g2gr\" (UID: \"df39047e-a25e-4661-a34e-a6bca0f344f2\") " pod="openshift-image-registry/node-ca-2g2gr"
Oct 30 23:06:55.548848 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.548828    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqgxb\" (UniqueName: \"kubernetes.io/projected/108c7917-29b6-47bc-a590-35fd94c3de19-kube-api-access-rqgxb\") pod \"node-resolver-gldcx\" (UID: \"108c7917-29b6-47bc-a590-35fd94c3de19\") " pod="openshift-dns/node-resolver-gldcx"
Oct 30 23:06:55.551132 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.551112    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzppf\" (UniqueName: \"kubernetes.io/projected/4f785b83-23af-45c1-975c-715bdebea140-kube-api-access-nzppf\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:55.551344 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.551320    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqblk\" (UniqueName: \"kubernetes.io/projected/91deced5-f22e-4148-98da-6a232be165c3-kube-api-access-kqblk\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.551430 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.551395    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzt62\" (UniqueName: \"kubernetes.io/projected/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-kube-api-access-hzt62\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:55.552214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.552198    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhw29\" (UniqueName: \"kubernetes.io/projected/2cd84d6c-d204-4e07-b591-9930cc9062a5-kube-api-access-qhw29\") pod \"downloads-6dd789c8b7-9vzv5\" (UID: \"2cd84d6c-d204-4e07-b591-9930cc9062a5\") " pod="openshift-console/downloads-6dd789c8b7-9vzv5"
Oct 30 23:06:55.553048 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.553031    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb2fp\" (UniqueName: \"kubernetes.io/projected/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-kube-api-access-pb2fp\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:06:55.554001 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.553793    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5p98\" (UniqueName: \"kubernetes.io/projected/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-kube-api-access-z5p98\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.554001 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.553932    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5zjd\" (UniqueName: \"kubernetes.io/projected/df39047e-a25e-4661-a34e-a6bca0f344f2-kube-api-access-p5zjd\") pod \"node-ca-2g2gr\" (UID: \"df39047e-a25e-4661-a34e-a6bca0f344f2\") " pod="openshift-image-registry/node-ca-2g2gr"
Oct 30 23:06:55.554001 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.553930    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngj6h\" (UniqueName: \"kubernetes.io/projected/cdee91d4-262d-4142-9340-4243df4586b6-kube-api-access-ngj6h\") pod \"migrator-66df5d496b-m896h\" (UID: \"cdee91d4-262d-4142-9340-4243df4586b6\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-m896h"
Oct 30 23:06:55.554001 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.553958    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2kbb\" (UniqueName: \"kubernetes.io/projected/9bd99d0d-1f0f-4926-b835-880a64193ded-kube-api-access-k2kbb\") pod \"periodic-gathering-7v4kl-bjfqg\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") " pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:06:55.554249 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.554226    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mdsg\" (UniqueName: \"kubernetes.io/projected/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf-kube-api-access-9mdsg\") pod \"iptables-alerter-dzklv\" (UID: \"29261ce8-6b07-4ea5-afa4-8500cc1c7bbf\") " pod="openshift-network-operator/iptables-alerter-dzklv"
Oct 30 23:06:55.554400 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.554383    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2fzc\" (UniqueName: \"kubernetes.io/projected/4e4bb115-b334-423e-86f1-1e1abb499b5d-kube-api-access-l2fzc\") pod \"dns-default-8qhk5\" (UID: \"4e4bb115-b334-423e-86f1-1e1abb499b5d\") " pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:55.554457 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.554407    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6bjr\" (UniqueName: \"kubernetes.io/projected/81b805c0-763e-4cd9-b6f5-af98767b298e-kube-api-access-d6bjr\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.554457 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.554413    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmhb8\" (UniqueName: \"kubernetes.io/projected/1918245f-b465-485f-9603-04ebe4846085-kube-api-access-pmhb8\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.555027 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.555009    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq95j\" (UniqueName: \"kubernetes.io/projected/58ec1adc-599b-4796-b12e-1cc85a834b15-kube-api-access-mq95j\") pod \"network-metrics-daemon-pln6h\" (UID: \"58ec1adc-599b-4796-b12e-1cc85a834b15\") " pod="openshift-multus/network-metrics-daemon-pln6h"
Oct 30 23:06:55.555577 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.555554    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqcxx\" (UniqueName: \"kubernetes.io/projected/06742bba-beca-46c9-8614-2fa5973c4864-kube-api-access-rqcxx\") pod \"service-ca-5fffc97bdf-9jlqk\" (UID: \"06742bba-beca-46c9-8614-2fa5973c4864\") " pod="openshift-service-ca/service-ca-5fffc97bdf-9jlqk"
Oct 30 23:06:55.555824 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.555795    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a07ae84d-1e2a-4f1e-a763-e5363ccdd520-bound-sa-token\") pod \"image-registry-7f5789fcdd-txvlr\" (UID: \"a07ae84d-1e2a-4f1e-a763-e5363ccdd520\") " pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:06:55.556013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.555990    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q46ww\" (UniqueName: \"kubernetes.io/projected/70a51653-2167-4e11-bf04-81f5b10c9c9d-kube-api-access-q46ww\") pod \"network-check-target-qvbgk\" (UID: \"70a51653-2167-4e11-bf04-81f5b10c9c9d\") " pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:06:55.556173 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.556157    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bm7q\" (UniqueName: \"kubernetes.io/projected/b99dd331-1d6d-434c-9b02-1b7694955006-kube-api-access-5bm7q\") pod \"ingress-canary-xslb9\" (UID: \"b99dd331-1d6d-434c-9b02-1b7694955006\") " pod="openshift-ingress-canary/ingress-canary-xslb9"
Oct 30 23:06:55.598627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598575    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-lib-modules\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.598627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598631    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-multus-cni-dir\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.598853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598650    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-etc-openvswitch\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.598853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598665    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-systemd-units\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.598853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598681    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-sysconfig\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.598853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598707    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-device-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.598853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598748    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-etc-openvswitch\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.598853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598752    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-lib-modules\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.598853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598771    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-hostroot\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.598853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598752    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-systemd-units\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.598853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598778    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-sysconfig\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.598853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598760    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-device-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.598853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598800    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e942185-68e3-4d57-9f42-49d2965b41e4-kubelet-config\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:55.598853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598809    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-multus-cni-dir\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.598853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598826    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xcsm\" (UniqueName: \"kubernetes.io/projected/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-kube-api-access-2xcsm\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.598853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598838    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e942185-68e3-4d57-9f42-49d2965b41e4-kubelet-config\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:55.598853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598847    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-system-cni-dir\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:06:55.598853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598861    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-slash\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598826    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-hostroot\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598894    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-run-netns\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598910    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-system-cni-dir\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598919    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df39047e-a25e-4661-a34e-a6bca0f344f2-host\") pod \"node-ca-2g2gr\" (UID: \"df39047e-a25e-4661-a34e-a6bca0f344f2\") " pod="openshift-image-registry/node-ca-2g2gr"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598926    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-slash\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598943    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-var-lib-cni-bin\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598946    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-run-netns\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598967    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df39047e-a25e-4661-a34e-a6bca0f344f2-host\") pod \"node-ca-2g2gr\" (UID: \"df39047e-a25e-4661-a34e-a6bca0f344f2\") " pod="openshift-image-registry/node-ca-2g2gr"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598967    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-var-lib-kubelet\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598985    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-var-lib-cni-bin\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598998    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-var-lib-kubelet\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.598998    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-etc-kubernetes\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599057    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-sysctl-conf\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599044    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-etc-kubernetes\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599079    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-modprobe-d\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599111    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f785b83-23af-45c1-975c-715bdebea140-crio-socket\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599136    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf-host-slash\") pod \"iptables-alerter-dzklv\" (UID: \"29261ce8-6b07-4ea5-afa4-8500cc1c7bbf\") " pod="openshift-network-operator/iptables-alerter-dzklv"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599182    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-run-ovn-kubernetes\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599195    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-sysctl-conf\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599213    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4f785b83-23af-45c1-975c-715bdebea140-crio-socket\") pod \"insights-runtime-extractor-tcf86\" (UID: \"4f785b83-23af-45c1-975c-715bdebea140\") " pod="openshift-insights/insights-runtime-extractor-tcf86"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599224    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/29261ce8-6b07-4ea5-afa4-8500cc1c7bbf-host-slash\") pod \"iptables-alerter-dzklv\" (UID: \"29261ce8-6b07-4ea5-afa4-8500cc1c7bbf\") " pod="openshift-network-operator/iptables-alerter-dzklv"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599234    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-sysctl-d\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599264    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-run-ovn-kubernetes\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599264    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-kubernetes\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599294    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-etc-selinux\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599321    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-kubernetes\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599324    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-modprobe-d\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599355    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-run-netns\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599333    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-run-netns\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599371    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-etc-selinux\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599377    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-sysctl-d\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599382    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e942185-68e3-4d57-9f42-49d2965b41e4-dbus\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599405    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-var-lib-openvswitch\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.599436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599434    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-var-lib-openvswitch\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599473    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e942185-68e3-4d57-9f42-49d2965b41e4-dbus\") pod \"global-pull-secret-syncer-cthfm\" (UID: \"2e942185-68e3-4d57-9f42-49d2965b41e4\") " pod="kube-system/global-pull-secret-syncer-cthfm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599499    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-registration-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599517    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-cnibin\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599540    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-node-exporter-accelerators-collector-config\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599565    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599614    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-systemd\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599629    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-cnibin\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599648    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-multus-socket-dir-parent\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599683    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-etc-systemd\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599715    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599616    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-registration-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599767    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-multus-socket-dir-parent\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599673    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-sys-fs\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599809    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-log-socket\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599859    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-sys\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599887    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-cni-bin\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599892    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-sys-fs\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599911    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-root\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599933    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-metrics-client-ca\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599951    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-sys\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599958    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-run\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599968    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-cni-bin\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599975    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-root\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.599993    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-log-socket\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600017    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-run\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600021    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-run-k8s-cni-cncf-io\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600066    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-node-log\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600095    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-node-exporter-tls\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600110    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-run-k8s-cni-cncf-io\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600140    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-node-log\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600173    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-node-exporter-textfile\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600207    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-node-exporter-accelerators-collector-config\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600232    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-socket-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600259    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-kubelet\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600284    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-multus-conf-dir\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600307    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-sys\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600309    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-socket-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600347    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-run-systemd\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600352    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-sys\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600372    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-run-openvswitch\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600374    5078 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600395    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-kubelet\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600399    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-run-ovn\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600433    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600445    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-run-systemd\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600448    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-node-exporter-textfile\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600462    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-system-cni-dir\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600467    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-metrics-client-ca\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600486    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-multus-conf-dir\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600489    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-run-multus-certs\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600492    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-run-openvswitch\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600518    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-run-ovn\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600528    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-system-cni-dir\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600545    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-host\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600529    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-run-multus-certs\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600573    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/108c7917-29b6-47bc-a590-35fd94c3de19-hosts-file\") pod \"node-resolver-gldcx\" (UID: \"108c7917-29b6-47bc-a590-35fd94c3de19\") " pod="openshift-dns/node-resolver-gldcx"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600612    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-host\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600618    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600676    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-cnibin\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600703    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-var-lib-cni-multus\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600722    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-cnibin\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600727    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-cni-netd\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600661    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/108c7917-29b6-47bc-a590-35fd94c3de19-hosts-file\") pod \"node-resolver-gldcx\" (UID: \"108c7917-29b6-47bc-a590-35fd94c3de19\") " pod="openshift-dns/node-resolver-gldcx"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600735    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6d5aef87-7d40-4aa2-958a-1fb73d427e3b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-z7d8k\" (UID: \"6d5aef87-7d40-4aa2-958a-1fb73d427e3b\") " pod="openshift-multus/multus-additional-cni-plugins-z7d8k"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600751    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-node-exporter-wtmp\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600761    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/91deced5-f22e-4148-98da-6a232be165c3-host-var-lib-cni-multus\") pod \"multus-wk66h\" (UID: \"91deced5-f22e-4148-98da-6a232be165c3\") " pod="openshift-multus/multus-wk66h"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600795    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/22aabe7f-74b4-4cc7-b88b-1ec78aecc925-host-cni-netd\") pod \"ovnkube-node-4c5rm\" (UID: \"22aabe7f-74b4-4cc7-b88b-1ec78aecc925\") " pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600796    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-var-lib-kubelet\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600826    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.600821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600853    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-node-exporter-wtmp\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.604044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600889    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81b805c0-763e-4cd9-b6f5-af98767b298e-var-lib-kubelet\") pod \"tuned-rcgrg\" (UID: \"81b805c0-763e-4cd9-b6f5-af98767b298e\") " pod="openshift-cluster-node-tuning-operator/tuned-rcgrg"
Oct 30 23:06:55.604044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.600893    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1918245f-b465-485f-9603-04ebe4846085-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2lw79\" (UID: \"1918245f-b465-485f-9603-04ebe4846085\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2lw79"
Oct 30 23:06:55.604044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.602890    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-node-exporter-tls\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.604044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.602983    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.659674 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.659643    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xcsm\" (UniqueName: \"kubernetes.io/projected/589b7d5b-70f8-440e-a171-3ba10ed5bf7a-kube-api-access-2xcsm\") pod \"node-exporter-6bv5g\" (UID: \"589b7d5b-70f8-440e-a171-3ba10ed5bf7a\") " pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.682497 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.682476    5078 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:06:55.696903 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:55.696870    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6bv5g"
Oct 30 23:06:55.707862 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:06:55.707833    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589b7d5b_70f8_440e_a171_3ba10ed5bf7a.slice/crio-7c5db51dd2b30b0f51f6126cf77cc13fc395cb0383eacd416f114f898789181d WatchSource:0}: Error finding container 7c5db51dd2b30b0f51f6126cf77cc13fc395cb0383eacd416f114f898789181d: Status 404 returned error can't find the container with id 7c5db51dd2b30b0f51f6126cf77cc13fc395cb0383eacd416f114f898789181d
Oct 30 23:06:56.320382 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:06:56.320343    5078 log.go:32] "PullImage from image service failed" err="rpc error: code = Canceled desc = unable to pull image: copying system image from manifest list: copying config: context canceled" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:36c3da5821b9e51334e9c523aca7fca6a9ef8b853fd4f238b908b086b129619d"
Oct 30 23:06:56.320811 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:06:56.320536    5078 kuberuntime_manager.go:1707] "Unhandled Error" err="container download-server start failed in pod downloads-6dd789c8b7-9vzv5_openshift-console(2cd84d6c-d204-4e07-b591-9930cc9062a5): ErrImagePull: rpc error: code = Canceled desc = unable to pull image: copying system image from manifest list: copying config: context canceled" logger="UnhandledError"
Oct 30 23:06:56.320811 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:06:56.320564    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"download-server\" with ErrImagePull: \"rpc error: code = Canceled desc = unable to pull image: copying system image from manifest list: copying config: context canceled\"" pod="openshift-console/downloads-6dd789c8b7-9vzv5" podUID="2cd84d6c-d204-4e07-b591-9930cc9062a5"
Oct 30 23:06:56.371104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:56.371072    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6bv5g" event={"ID":"589b7d5b-70f8-440e-a171-3ba10ed5bf7a","Type":"ContainerStarted","Data":"7c5db51dd2b30b0f51f6126cf77cc13fc395cb0383eacd416f114f898789181d"}
Oct 30 23:06:57.376542 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:57.376506    5078 generic.go:356] "Generic (PLEG): container finished" podID="589b7d5b-70f8-440e-a171-3ba10ed5bf7a" containerID="aa879c88d8d28af81093d83038fd11eaafeb88c72e1b66b623bb025efe134dc6" exitCode=0
Oct 30 23:06:57.376946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:57.376557    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6bv5g" event={"ID":"589b7d5b-70f8-440e-a171-3ba10ed5bf7a","Type":"ContainerDied","Data":"aa879c88d8d28af81093d83038fd11eaafeb88c72e1b66b623bb025efe134dc6"}
Oct 30 23:06:57.563705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:57.563675    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-fg8fg"
Oct 30 23:06:57.929473 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:57.929434    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:06:57.936783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:57.936757    5078 patch_prober.go:29] interesting pod/image-registry-7f5789fcdd-txvlr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 30 23:06:57.936917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:57.936800    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr" podUID="a07ae84d-1e2a-4f1e-a763-e5363ccdd520" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:06:58.381314 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:58.381276    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6bv5g" event={"ID":"589b7d5b-70f8-440e-a171-3ba10ed5bf7a","Type":"ContainerStarted","Data":"4fea7c7d3262867ef1c4ef35e4660e6ead7477e00e671fa68096e0de495c7d71"}
Oct 30 23:06:58.381738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:58.381338    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6bv5g" event={"ID":"589b7d5b-70f8-440e-a171-3ba10ed5bf7a","Type":"ContainerStarted","Data":"4c1834481126d953114bf408f34e5c685c7960d65c37cded48a13cd65f469447"}
Oct 30 23:06:58.499808 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:06:58.499751    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6bv5g" podStartSLOduration=2.00211817 podStartE2EDuration="3.499736396s" totalImagesPullingTime="1.422607039s" totalInitContainerRuntime="75.011187ms" isStatefulPod=true podCreationTimestamp="2026-10-30 23:06:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:06:58.499361604 +0000 UTC m=+4.387267992" watchObservedRunningTime="2026-10-30 23:06:58.499736396 +0000 UTC m=+4.387642748"
Oct 30 23:07:00.932151 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:00.932124    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8qhk5"
Oct 30 23:07:01.229016 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:01.228938    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:07:03.176495 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:03.176464    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qvbgk"
Oct 30 23:07:04.993367 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:04.993340    5078 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-205.us-west-1.compute.internal" event="NodeReady"
Oct 30 23:07:05.165849 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:05.165824    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:07:05.166146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:05.166129    5078 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 23:07:07.937546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:07.937521    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f5789fcdd-txvlr"
Oct 30 23:07:10.423077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:10.422999    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-9vzv5" event={"ID":"2cd84d6c-d204-4e07-b591-9930cc9062a5","Type":"ContainerStarted","Data":"9523883fbee6a080e435829539f1b0870edfaf116f62cf8dc6734284e406ca6d"}
Oct 30 23:07:10.423527 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:10.423486    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-9vzv5"
Oct 30 23:07:10.439105 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:10.439076    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-9vzv5"
Oct 30 23:07:14.494144 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:14.494108    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-4c5rm"
Oct 30 23:07:17.596228 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:07:17.596188    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:07:47.588903 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:07:47.588857    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:07:52.362820 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:52.362761    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-7v4kl-bjfqg"]
Oct 30 23:07:52.363214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:52.363089    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-7v4kl-bjfqg" podUID="9bd99d0d-1f0f-4926-b835-880a64193ded" containerName="insights-gathering" containerID="cri-o://f20253b85443155ced8d865281bbea235538ce29ff6c844d3d4762a7ddbdcd65" gracePeriod=30
Oct 30 23:07:52.561909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:52.561880    5078 generic.go:356] "Generic (PLEG): container finished" podID="9bd99d0d-1f0f-4926-b835-880a64193ded" containerID="f20253b85443155ced8d865281bbea235538ce29ff6c844d3d4762a7ddbdcd65" exitCode=2
Oct 30 23:07:52.562047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:52.561950    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7v4kl-bjfqg" event={"ID":"9bd99d0d-1f0f-4926-b835-880a64193ded","Type":"ContainerDied","Data":"f20253b85443155ced8d865281bbea235538ce29ff6c844d3d4762a7ddbdcd65"}
Oct 30 23:07:52.634810 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:52.634749    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:07:52.704783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:52.704759    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-trusted-ca-bundle\") pod \"9bd99d0d-1f0f-4926-b835-880a64193ded\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") "
Oct 30 23:07:52.704868 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:52.704805    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9bd99d0d-1f0f-4926-b835-880a64193ded-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9bd99d0d-1f0f-4926-b835-880a64193ded-archives-path\") pod \"9bd99d0d-1f0f-4926-b835-880a64193ded\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") "
Oct 30 23:07:52.704868 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:52.704843    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd99d0d-1f0f-4926-b835-880a64193ded-kube-api-access-k2kbb\" (UniqueName: \"kubernetes.io/projected/9bd99d0d-1f0f-4926-b835-880a64193ded-kube-api-access-k2kbb\") pod \"9bd99d0d-1f0f-4926-b835-880a64193ded\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") "
Oct 30 23:07:52.704868 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:52.704863    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-service-ca-bundle\") pod \"9bd99d0d-1f0f-4926-b835-880a64193ded\" (UID: \"9bd99d0d-1f0f-4926-b835-880a64193ded\") "
Oct 30 23:07:52.705160 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:52.705135    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9bd99d0d-1f0f-4926-b835-880a64193ded-archives-path" pod "9bd99d0d-1f0f-4926-b835-880a64193ded" (UID: "9bd99d0d-1f0f-4926-b835-880a64193ded"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:07:52.705236 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:52.705157    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-trusted-ca-bundle" pod "9bd99d0d-1f0f-4926-b835-880a64193ded" (UID: "9bd99d0d-1f0f-4926-b835-880a64193ded"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:07:52.705236 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:52.705206    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-service-ca-bundle" pod "9bd99d0d-1f0f-4926-b835-880a64193ded" (UID: "9bd99d0d-1f0f-4926-b835-880a64193ded"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:07:52.707257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:52.707237    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd99d0d-1f0f-4926-b835-880a64193ded-kube-api-access-k2kbb" pod "9bd99d0d-1f0f-4926-b835-880a64193ded" (UID: "9bd99d0d-1f0f-4926-b835-880a64193ded"). InnerVolumeSpecName "kube-api-access-k2kbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:07:52.805847 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:52.805820    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2kbb\" (UniqueName: \"kubernetes.io/projected/9bd99d0d-1f0f-4926-b835-880a64193ded-kube-api-access-k2kbb\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:07:52.805847 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:52.805846    5078 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-service-ca-bundle\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:07:52.805998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:52.805857    5078 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bd99d0d-1f0f-4926-b835-880a64193ded-trusted-ca-bundle\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:07:52.805998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:52.805867    5078 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9bd99d0d-1f0f-4926-b835-880a64193ded-archives-path\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:07:53.566284 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:53.566249    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7v4kl-bjfqg"
Oct 30 23:07:53.566840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:53.566249    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7v4kl-bjfqg" event={"ID":"9bd99d0d-1f0f-4926-b835-880a64193ded","Type":"ContainerDied","Data":"f3d048a7dac8692c140c08e6cdf2c6de069f73e17c8625044406e1ad267865a0"}
Oct 30 23:07:53.566840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:53.566403    5078 scope.go:122] "RemoveContainer" containerID="f20253b85443155ced8d865281bbea235538ce29ff6c844d3d4762a7ddbdcd65"
Oct 30 23:07:53.728056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:53.728028    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-7v4kl-bjfqg"]
Oct 30 23:07:53.777577 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:53.777547    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-7v4kl-bjfqg"]
Oct 30 23:07:54.295797 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:07:54.295767    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bd99d0d-1f0f-4926-b835-880a64193ded" path="/var/lib/kubelet/pods/9bd99d0d-1f0f-4926-b835-880a64193ded/volumes"
Oct 30 23:08:17.589127 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:08:17.589089    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:08:47.589797 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:08:47.589718    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:09:17.592342 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:09:17.592307    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:09:47.588708 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:09:47.588669    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:10:17.589817 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:10:17.589777    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:10:47.590046 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:10:47.590010    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:11:17.589692 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:11:17.589659    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:11:26.565318 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.565287    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr"]
Oct 30 23:11:26.565802 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.565522    5078 cpu_manager.go:414] "containerMap: removing container" podUID="9bd99d0d-1f0f-4926-b835-880a64193ded" containerName="insights-gathering"
Oct 30 23:11:26.565802 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.565532    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd99d0d-1f0f-4926-b835-880a64193ded" containerName="insights-gathering"
Oct 30 23:11:26.565802 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.565573    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd99d0d-1f0f-4926-b835-880a64193ded" containerName="insights-gathering"
Oct 30 23:11:26.567117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.567102    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr"
Oct 30 23:11:26.669796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.669769    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-nn9z7\""
Oct 30 23:11:26.669796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.669776    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 23:11:26.669960 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.669816    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 23:11:26.674222 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.674193    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr"]
Oct 30 23:11:26.714827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.714798    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d3c6edf-0e9f-4d01-989d-015851f65985-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr\" (UID: \"5d3c6edf-0e9f-4d01-989d-015851f65985\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr"
Oct 30 23:11:26.714953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.714833    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wllx\" (UniqueName: \"kubernetes.io/projected/5d3c6edf-0e9f-4d01-989d-015851f65985-kube-api-access-6wllx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr\" (UID: \"5d3c6edf-0e9f-4d01-989d-015851f65985\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr"
Oct 30 23:11:26.714953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.714862    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d3c6edf-0e9f-4d01-989d-015851f65985-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr\" (UID: \"5d3c6edf-0e9f-4d01-989d-015851f65985\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr"
Oct 30 23:11:26.816162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.816085    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d3c6edf-0e9f-4d01-989d-015851f65985-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr\" (UID: \"5d3c6edf-0e9f-4d01-989d-015851f65985\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr"
Oct 30 23:11:26.816162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.816155    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d3c6edf-0e9f-4d01-989d-015851f65985-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr\" (UID: \"5d3c6edf-0e9f-4d01-989d-015851f65985\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr"
Oct 30 23:11:26.816376 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.816189    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wllx\" (UniqueName: \"kubernetes.io/projected/5d3c6edf-0e9f-4d01-989d-015851f65985-kube-api-access-6wllx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr\" (UID: \"5d3c6edf-0e9f-4d01-989d-015851f65985\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr"
Oct 30 23:11:26.816548 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.816526    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d3c6edf-0e9f-4d01-989d-015851f65985-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr\" (UID: \"5d3c6edf-0e9f-4d01-989d-015851f65985\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr"
Oct 30 23:11:26.816618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.816532    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d3c6edf-0e9f-4d01-989d-015851f65985-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr\" (UID: \"5d3c6edf-0e9f-4d01-989d-015851f65985\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr"
Oct 30 23:11:26.873514 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.873478    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wllx\" (UniqueName: \"kubernetes.io/projected/5d3c6edf-0e9f-4d01-989d-015851f65985-kube-api-access-6wllx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr\" (UID: \"5d3c6edf-0e9f-4d01-989d-015851f65985\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr"
Oct 30 23:11:26.878322 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:26.878293    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr"
Oct 30 23:11:27.120062 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.119996    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"]
Oct 30 23:11:27.121850 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.121836    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.182136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.182092    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 23:11:27.182136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.182118    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 23:11:27.182136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.182107    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fdjft\""
Oct 30 23:11:27.182364 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.182107    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 23:11:27.182364 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.182101    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 23:11:27.182364 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.182127    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 23:11:27.182364 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.182187    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 23:11:27.189628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.189582    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"]
Oct 30 23:11:27.220116 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.220085    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89grx\" (UniqueName: \"kubernetes.io/projected/af0356e9-fe3e-4123-9be2-41f76c02c78b-kube-api-access-89grx\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.220116 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.220116    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.220306 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.220143    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.220306 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.220222    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.220306 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.220253    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.220414 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.220319    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.220414 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.220347    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0356e9-fe3e-4123-9be2-41f76c02c78b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.248612 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.248567    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr"]
Oct 30 23:11:27.252414 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:11:27.252390    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3c6edf_0e9f_4d01_989d_015851f65985.slice/crio-aa25d54a638a3da2c8b36499f5739eaa194a9089a58e28eee8c918465de65214 WatchSource:0}: Error finding container aa25d54a638a3da2c8b36499f5739eaa194a9089a58e28eee8c918465de65214: Status 404 returned error can't find the container with id aa25d54a638a3da2c8b36499f5739eaa194a9089a58e28eee8c918465de65214
Oct 30 23:11:27.321503 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.321463    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.321503 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.321515    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.321752 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.321629    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.321752 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.321692    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.321752 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.321726    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.321898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.321756    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0356e9-fe3e-4123-9be2-41f76c02c78b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.321948 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.321904    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89grx\" (UniqueName: \"kubernetes.io/projected/af0356e9-fe3e-4123-9be2-41f76c02c78b-kube-api-access-89grx\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.322536 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.322379    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.324119 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.324094    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0356e9-fe3e-4123-9be2-41f76c02c78b-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.324119 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.324109    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.324260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.324174    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.324297 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.324268    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.380117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.380045    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.380117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.380058    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89grx\" (UniqueName: \"kubernetes.io/projected/af0356e9-fe3e-4123-9be2-41f76c02c78b-kube-api-access-89grx\") pod \"istiod-openshift-gateway-7876bfcb74-xz7tp\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.431748 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.431707    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:27.847307 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:27.847267    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"]
Oct 30 23:11:27.850710 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:11:27.850681    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf0356e9_fe3e_4123_9be2_41f76c02c78b.slice/crio-c7820797cd06b9ef75bc6c07c022297bcca1c605243bd623a6134b533bed2d21 WatchSource:0}: Error finding container c7820797cd06b9ef75bc6c07c022297bcca1c605243bd623a6134b533bed2d21: Status 404 returned error can't find the container with id c7820797cd06b9ef75bc6c07c022297bcca1c605243bd623a6134b533bed2d21
Oct 30 23:11:28.206524 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:28.206438    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp" event={"ID":"af0356e9-fe3e-4123-9be2-41f76c02c78b","Type":"ContainerStarted","Data":"c7820797cd06b9ef75bc6c07c022297bcca1c605243bd623a6134b533bed2d21"}
Oct 30 23:11:28.207517 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:28.207493    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr" event={"ID":"5d3c6edf-0e9f-4d01-989d-015851f65985","Type":"ContainerStarted","Data":"aa25d54a638a3da2c8b36499f5739eaa194a9089a58e28eee8c918465de65214"}
Oct 30 23:11:31.913705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:31.913665    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:11:31.914062 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:31.913744    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:11:32.226545 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:32.226461    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp" event={"ID":"af0356e9-fe3e-4123-9be2-41f76c02c78b","Type":"ContainerStarted","Data":"f7f447eacdca39bff606b888344b94e3e4c36534185b9df5851916f93d26d4d9"}
Oct 30 23:11:32.226728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:32.226652    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:32.228494 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:32.228453    5078 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-xz7tp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:11:32.228680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:32.228509    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp" podUID="af0356e9-fe3e-4123-9be2-41f76c02c78b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:11:32.433182 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:32.433121    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp" podStartSLOduration=1.372961998 podStartE2EDuration="5.433101631s" totalImagesPullingTime="4.060139633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:11:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:11:32.393362538 +0000 UTC m=+278.281268902" watchObservedRunningTime="2026-10-30 23:11:32.433101631 +0000 UTC m=+278.321007986"
Oct 30 23:11:33.231186 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:33.231104    5078 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-xz7tp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:11:33.231186 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:33.231165    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp" podUID="af0356e9-fe3e-4123-9be2-41f76c02c78b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:11:35.238879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:35.238845    5078 generic.go:356] "Generic (PLEG): container finished" podID="5d3c6edf-0e9f-4d01-989d-015851f65985" containerID="167dde5dbda3105f6ab674da9abfbfe1542b4eb743858141c54576ce8163e971" exitCode=0
Oct 30 23:11:35.239262 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:35.238936    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr" event={"ID":"5d3c6edf-0e9f-4d01-989d-015851f65985","Type":"ContainerDied","Data":"167dde5dbda3105f6ab674da9abfbfe1542b4eb743858141c54576ce8163e971"}
Oct 30 23:11:36.232202 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:36.232174    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:11:39.254608 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:39.254560    5078 generic.go:356] "Generic (PLEG): container finished" podID="5d3c6edf-0e9f-4d01-989d-015851f65985" containerID="59a7181d1c9e9e27a31051e8404feef594123e90ce8ddcb2568f34a7b218e761" exitCode=0
Oct 30 23:11:39.255076 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:39.254651    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr" event={"ID":"5d3c6edf-0e9f-4d01-989d-015851f65985","Type":"ContainerDied","Data":"59a7181d1c9e9e27a31051e8404feef594123e90ce8ddcb2568f34a7b218e761"}
Oct 30 23:11:46.280714 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:46.280672    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr" event={"ID":"5d3c6edf-0e9f-4d01-989d-015851f65985","Type":"ContainerStarted","Data":"b58c98b60221c9575056f54afbc372b1d490cc81a3ff3642631fa9456f594dcf"}
Oct 30 23:11:46.395200 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:46.395146    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr" podStartSLOduration=3.191530514 podStartE2EDuration="20.39513043s" totalImagesPullingTime="17.203599916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:11:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:11:46.394515836 +0000 UTC m=+292.282422186" watchObservedRunningTime="2026-10-30 23:11:46.39513043 +0000 UTC m=+292.283036785"
Oct 30 23:11:47.286214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:47.286180    5078 generic.go:356] "Generic (PLEG): container finished" podID="5d3c6edf-0e9f-4d01-989d-015851f65985" containerID="b58c98b60221c9575056f54afbc372b1d490cc81a3ff3642631fa9456f594dcf" exitCode=0
Oct 30 23:11:47.286630 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:47.286243    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr" event={"ID":"5d3c6edf-0e9f-4d01-989d-015851f65985","Type":"ContainerDied","Data":"b58c98b60221c9575056f54afbc372b1d490cc81a3ff3642631fa9456f594dcf"}
Oct 30 23:11:47.589299 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:11:47.589266    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:11:48.430996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:48.430968    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr"
Oct 30 23:11:48.589508 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:48.589477    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d3c6edf-0e9f-4d01-989d-015851f65985-util\" (UniqueName: \"kubernetes.io/empty-dir/5d3c6edf-0e9f-4d01-989d-015851f65985-util\") pod \"5d3c6edf-0e9f-4d01-989d-015851f65985\" (UID: \"5d3c6edf-0e9f-4d01-989d-015851f65985\") "
Oct 30 23:11:48.589728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:48.589535    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3c6edf-0e9f-4d01-989d-015851f65985-kube-api-access-6wllx\" (UniqueName: \"kubernetes.io/projected/5d3c6edf-0e9f-4d01-989d-015851f65985-kube-api-access-6wllx\") pod \"5d3c6edf-0e9f-4d01-989d-015851f65985\" (UID: \"5d3c6edf-0e9f-4d01-989d-015851f65985\") "
Oct 30 23:11:48.589728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:48.589606    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d3c6edf-0e9f-4d01-989d-015851f65985-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d3c6edf-0e9f-4d01-989d-015851f65985-bundle\") pod \"5d3c6edf-0e9f-4d01-989d-015851f65985\" (UID: \"5d3c6edf-0e9f-4d01-989d-015851f65985\") "
Oct 30 23:11:48.590257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:48.590170    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3c6edf-0e9f-4d01-989d-015851f65985-bundle" pod "5d3c6edf-0e9f-4d01-989d-015851f65985" (UID: "5d3c6edf-0e9f-4d01-989d-015851f65985"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:11:48.592046 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:48.592021    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3c6edf-0e9f-4d01-989d-015851f65985-kube-api-access-6wllx" pod "5d3c6edf-0e9f-4d01-989d-015851f65985" (UID: "5d3c6edf-0e9f-4d01-989d-015851f65985"). InnerVolumeSpecName "kube-api-access-6wllx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:11:48.593562 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:48.593537    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d3c6edf-0e9f-4d01-989d-015851f65985-util" pod "5d3c6edf-0e9f-4d01-989d-015851f65985" (UID: "5d3c6edf-0e9f-4d01-989d-015851f65985"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:11:48.690664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:48.690634    5078 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5d3c6edf-0e9f-4d01-989d-015851f65985-bundle\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:11:48.690664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:48.690657    5078 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5d3c6edf-0e9f-4d01-989d-015851f65985-util\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:11:48.690664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:48.690667    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wllx\" (UniqueName: \"kubernetes.io/projected/5d3c6edf-0e9f-4d01-989d-015851f65985-kube-api-access-6wllx\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:11:49.294786 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:49.294752    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr" event={"ID":"5d3c6edf-0e9f-4d01-989d-015851f65985","Type":"ContainerDied","Data":"aa25d54a638a3da2c8b36499f5739eaa194a9089a58e28eee8c918465de65214"}
Oct 30 23:11:49.294945 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:49.294795    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa25d54a638a3da2c8b36499f5739eaa194a9089a58e28eee8c918465de65214"
Oct 30 23:11:49.294945 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:11:49.294765    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfzrsr"
Oct 30 23:12:17.589448 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:12:17.589414    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:12:24.510695 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.510666    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"]
Oct 30 23:12:24.511295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.511118    5078 cpu_manager.go:414] "containerMap: removing container" podUID="5d3c6edf-0e9f-4d01-989d-015851f65985" containerName="util"
Oct 30 23:12:24.511295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.511137    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3c6edf-0e9f-4d01-989d-015851f65985" containerName="util"
Oct 30 23:12:24.511295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.511169    5078 cpu_manager.go:414] "containerMap: removing container" podUID="5d3c6edf-0e9f-4d01-989d-015851f65985" containerName="extract"
Oct 30 23:12:24.511295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.511176    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3c6edf-0e9f-4d01-989d-015851f65985" containerName="extract"
Oct 30 23:12:24.511295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.511190    5078 cpu_manager.go:414] "containerMap: removing container" podUID="5d3c6edf-0e9f-4d01-989d-015851f65985" containerName="pull"
Oct 30 23:12:24.511295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.511197    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3c6edf-0e9f-4d01-989d-015851f65985" containerName="pull"
Oct 30 23:12:24.511295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.511273    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3c6edf-0e9f-4d01-989d-015851f65985" containerName="extract"
Oct 30 23:12:24.513237 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.513217    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.539091 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.539066    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knv2r\" (UniqueName: \"kubernetes.io/projected/3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7-kube-api-access-knv2r\") pod \"opendatahub-operator-controller-manager-5b8f94678b-7pf24\" (UID: \"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.539204 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.539094    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b8f94678b-7pf24\" (UID: \"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.539204 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.539114    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b8f94678b-7pf24\" (UID: \"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.539295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.539274    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7-tmp\") pod \"opendatahub-operator-controller-manager-5b8f94678b-7pf24\" (UID: \"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.539340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.539307    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b8f94678b-7pf24\" (UID: \"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.539340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.539323    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7-manifests\") pod \"opendatahub-operator-controller-manager-5b8f94678b-7pf24\" (UID: \"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.630146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.630118    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 23:12:24.630349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.630331    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 23:12:24.633651 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.630809    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-zt2dv\""
Oct 30 23:12:24.633651 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.630872    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 23:12:24.633651 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.632584    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 23:12:24.633651 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.632676    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 23:12:24.639700 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.639678    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7-tmp\") pod \"opendatahub-operator-controller-manager-5b8f94678b-7pf24\" (UID: \"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.639821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.639710    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b8f94678b-7pf24\" (UID: \"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.639821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.639726    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7-manifests\") pod \"opendatahub-operator-controller-manager-5b8f94678b-7pf24\" (UID: \"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.639821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.639751    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knv2r\" (UniqueName: \"kubernetes.io/projected/3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7-kube-api-access-knv2r\") pod \"opendatahub-operator-controller-manager-5b8f94678b-7pf24\" (UID: \"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.639821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.639793    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b8f94678b-7pf24\" (UID: \"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.639821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.639819    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b8f94678b-7pf24\" (UID: \"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.640574 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.640552    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7-manifests\") pod \"opendatahub-operator-controller-manager-5b8f94678b-7pf24\" (UID: \"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.647315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.647292    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7-tmp\") pod \"opendatahub-operator-controller-manager-5b8f94678b-7pf24\" (UID: \"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.647512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.647491    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7-webhook-cert\") pod \"opendatahub-operator-controller-manager-5b8f94678b-7pf24\" (UID: \"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.647627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.647591    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5b8f94678b-7pf24\" (UID: \"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.647953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.647933    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7-metrics-cert\") pod \"opendatahub-operator-controller-manager-5b8f94678b-7pf24\" (UID: \"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.648341 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.648325    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"]
Oct 30 23:12:24.699783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.699753    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knv2r\" (UniqueName: \"kubernetes.io/projected/3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7-kube-api-access-knv2r\") pod \"opendatahub-operator-controller-manager-5b8f94678b-7pf24\" (UID: \"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:24.824228 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:24.824194    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:25.332095 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:25.331967    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"]
Oct 30 23:12:25.334779 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:12:25.334747    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c90f2bb_c7d1_4b63_8f69_e5f05c2f0bc7.slice/crio-a93514f47f36443e696c3ea858807c87a84f8e422bc2212080dec67bcd9272b5 WatchSource:0}: Error finding container a93514f47f36443e696c3ea858807c87a84f8e422bc2212080dec67bcd9272b5: Status 404 returned error can't find the container with id a93514f47f36443e696c3ea858807c87a84f8e422bc2212080dec67bcd9272b5
Oct 30 23:12:25.336320 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:25.336304    5078 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:12:25.406509 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:25.406476    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24" event={"ID":"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7","Type":"ContainerStarted","Data":"a93514f47f36443e696c3ea858807c87a84f8e422bc2212080dec67bcd9272b5"}
Oct 30 23:12:30.426071 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:30.426034    5078 generic.go:356] "Generic (PLEG): container finished" podID="3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7" containerID="0dc980938a5b031e248ba9d0522f9bf219f764c206d5fed771be3a371546acd8" exitCode=0
Oct 30 23:12:30.426582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:30.426087    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24" event={"ID":"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7","Type":"ContainerDied","Data":"0dc980938a5b031e248ba9d0522f9bf219f764c206d5fed771be3a371546acd8"}
Oct 30 23:12:31.431216 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:31.431181    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24" event={"ID":"3c90f2bb-c7d1-4b63-8f69-e5f05c2f0bc7","Type":"ContainerStarted","Data":"b8ddde50e567d3da76c5e310222d4e0048abc79552ba3775701a90ee0fb28c82"}
Oct 30 23:12:31.431765 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:31.431737    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:31.670830 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:31.670779    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24" podStartSLOduration=2.049073183 podStartE2EDuration="7.670765037s" totalImagesPullingTime="5.126598536s" totalInitContainerRuntime="495.093318ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:12:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:12:31.620900487 +0000 UTC m=+337.508806840" watchObservedRunningTime="2026-10-30 23:12:31.670765037 +0000 UTC m=+337.558671389"
Oct 30 23:12:43.439864 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:12:43.439835    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-5b8f94678b-7pf24"
Oct 30 23:12:47.590151 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:12:47.590119    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:13:01.246741 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:01.246708    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-bd9cf9d49-6tcdv"]
Oct 30 23:13:01.301997 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:01.301963    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-bd9cf9d49-6tcdv"
Oct 30 23:13:01.315549 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:01.315526    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-bd9cf9d49-6tcdv"]
Oct 30 23:13:01.413568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:01.413540    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8c68f95-3c84-4af1-9d26-8eedf2fc8f79-tmp\") pod \"kube-auth-proxy-bd9cf9d49-6tcdv\" (UID: \"a8c68f95-3c84-4af1-9d26-8eedf2fc8f79\") " pod="openshift-ingress/kube-auth-proxy-bd9cf9d49-6tcdv"
Oct 30 23:13:01.413781 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:01.413576    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78sbk\" (UniqueName: \"kubernetes.io/projected/a8c68f95-3c84-4af1-9d26-8eedf2fc8f79-kube-api-access-78sbk\") pod \"kube-auth-proxy-bd9cf9d49-6tcdv\" (UID: \"a8c68f95-3c84-4af1-9d26-8eedf2fc8f79\") " pod="openshift-ingress/kube-auth-proxy-bd9cf9d49-6tcdv"
Oct 30 23:13:01.413781 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:01.413669    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8c68f95-3c84-4af1-9d26-8eedf2fc8f79-tls-certs\") pod \"kube-auth-proxy-bd9cf9d49-6tcdv\" (UID: \"a8c68f95-3c84-4af1-9d26-8eedf2fc8f79\") " pod="openshift-ingress/kube-auth-proxy-bd9cf9d49-6tcdv"
Oct 30 23:13:01.451165 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:01.451136    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 23:13:01.470462 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:01.470437    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 23:13:01.470462 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:01.470443    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-xq6np\""
Oct 30 23:13:01.514853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:01.514783    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8c68f95-3c84-4af1-9d26-8eedf2fc8f79-tls-certs\") pod \"kube-auth-proxy-bd9cf9d49-6tcdv\" (UID: \"a8c68f95-3c84-4af1-9d26-8eedf2fc8f79\") " pod="openshift-ingress/kube-auth-proxy-bd9cf9d49-6tcdv"
Oct 30 23:13:01.515024 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:01.514849    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8c68f95-3c84-4af1-9d26-8eedf2fc8f79-tmp\") pod \"kube-auth-proxy-bd9cf9d49-6tcdv\" (UID: \"a8c68f95-3c84-4af1-9d26-8eedf2fc8f79\") " pod="openshift-ingress/kube-auth-proxy-bd9cf9d49-6tcdv"
Oct 30 23:13:01.515024 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:01.514891    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78sbk\" (UniqueName: \"kubernetes.io/projected/a8c68f95-3c84-4af1-9d26-8eedf2fc8f79-kube-api-access-78sbk\") pod \"kube-auth-proxy-bd9cf9d49-6tcdv\" (UID: \"a8c68f95-3c84-4af1-9d26-8eedf2fc8f79\") " pod="openshift-ingress/kube-auth-proxy-bd9cf9d49-6tcdv"
Oct 30 23:13:01.517137 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:01.517111    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a8c68f95-3c84-4af1-9d26-8eedf2fc8f79-tmp\") pod \"kube-auth-proxy-bd9cf9d49-6tcdv\" (UID: \"a8c68f95-3c84-4af1-9d26-8eedf2fc8f79\") " pod="openshift-ingress/kube-auth-proxy-bd9cf9d49-6tcdv"
Oct 30 23:13:01.517327 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:01.517311    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8c68f95-3c84-4af1-9d26-8eedf2fc8f79-tls-certs\") pod \"kube-auth-proxy-bd9cf9d49-6tcdv\" (UID: \"a8c68f95-3c84-4af1-9d26-8eedf2fc8f79\") " pod="openshift-ingress/kube-auth-proxy-bd9cf9d49-6tcdv"
Oct 30 23:13:01.576042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:01.576016    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78sbk\" (UniqueName: \"kubernetes.io/projected/a8c68f95-3c84-4af1-9d26-8eedf2fc8f79-kube-api-access-78sbk\") pod \"kube-auth-proxy-bd9cf9d49-6tcdv\" (UID: \"a8c68f95-3c84-4af1-9d26-8eedf2fc8f79\") " pod="openshift-ingress/kube-auth-proxy-bd9cf9d49-6tcdv"
Oct 30 23:13:01.626042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:01.626014    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-bd9cf9d49-6tcdv"
Oct 30 23:13:02.015939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:02.015912    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-bd9cf9d49-6tcdv"]
Oct 30 23:13:02.018318 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:13:02.018287    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c68f95_3c84_4af1_9d26_8eedf2fc8f79.slice/crio-32498122f1b8eaef0ffc8911bd307389e48692398e447069adc69b2b31545a32 WatchSource:0}: Error finding container 32498122f1b8eaef0ffc8911bd307389e48692398e447069adc69b2b31545a32: Status 404 returned error can't find the container with id 32498122f1b8eaef0ffc8911bd307389e48692398e447069adc69b2b31545a32
Oct 30 23:13:02.536754 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:02.536726    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-bd9cf9d49-6tcdv" event={"ID":"a8c68f95-3c84-4af1-9d26-8eedf2fc8f79","Type":"ContainerStarted","Data":"32498122f1b8eaef0ffc8911bd307389e48692398e447069adc69b2b31545a32"}
Oct 30 23:13:05.550492 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:05.550454    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-bd9cf9d49-6tcdv" event={"ID":"a8c68f95-3c84-4af1-9d26-8eedf2fc8f79","Type":"ContainerStarted","Data":"e78992318b3d0a00740856e463abe928c4f5e3441858fd483b9e85211751ac44"}
Oct 30 23:13:05.663966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:05.663904    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-bd9cf9d49-6tcdv" podStartSLOduration=1.8782450320000001 podStartE2EDuration="4.663886187s" totalImagesPullingTime="2.785641155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:13:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:13:05.663322446 +0000 UTC m=+371.551228796" watchObservedRunningTime="2026-10-30 23:13:05.663886187 +0000 UTC m=+371.551792545"
Oct 30 23:13:09.175338 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.175305    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"]
Oct 30 23:13:09.178546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.178529    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"
Oct 30 23:13:09.231162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.231133    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 23:13:09.231162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.231134    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 30 23:13:09.275667 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.275637    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2d7bf55-7997-4fe9-82a4-192f0c946deb-metrics-certs\") pod \"jobset-controller-manager-6d6b4cf5db-c9j8t\" (UID: \"a2d7bf55-7997-4fe9-82a4-192f0c946deb\") " pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"
Oct 30 23:13:09.275851 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.275688    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2d7bf55-7997-4fe9-82a4-192f0c946deb-manager-config\") pod \"jobset-controller-manager-6d6b4cf5db-c9j8t\" (UID: \"a2d7bf55-7997-4fe9-82a4-192f0c946deb\") " pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"
Oct 30 23:13:09.275851 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.275757    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpmh8\" (UniqueName: \"kubernetes.io/projected/a2d7bf55-7997-4fe9-82a4-192f0c946deb-kube-api-access-rpmh8\") pod \"jobset-controller-manager-6d6b4cf5db-c9j8t\" (UID: \"a2d7bf55-7997-4fe9-82a4-192f0c946deb\") " pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"
Oct 30 23:13:09.275851 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.275787    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2d7bf55-7997-4fe9-82a4-192f0c946deb-cert\") pod \"jobset-controller-manager-6d6b4cf5db-c9j8t\" (UID: \"a2d7bf55-7997-4fe9-82a4-192f0c946deb\") " pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"
Oct 30 23:13:09.280553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.280521    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 23:13:09.280553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.280531    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 30 23:13:09.280744 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.280572    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-sjkcr\""
Oct 30 23:13:09.280744 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.280618    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 30 23:13:09.281795 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.281776    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"]
Oct 30 23:13:09.376381 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.376347    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2d7bf55-7997-4fe9-82a4-192f0c946deb-metrics-certs\") pod \"jobset-controller-manager-6d6b4cf5db-c9j8t\" (UID: \"a2d7bf55-7997-4fe9-82a4-192f0c946deb\") " pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"
Oct 30 23:13:09.376575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.376396    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2d7bf55-7997-4fe9-82a4-192f0c946deb-manager-config\") pod \"jobset-controller-manager-6d6b4cf5db-c9j8t\" (UID: \"a2d7bf55-7997-4fe9-82a4-192f0c946deb\") " pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"
Oct 30 23:13:09.376575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.376425    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpmh8\" (UniqueName: \"kubernetes.io/projected/a2d7bf55-7997-4fe9-82a4-192f0c946deb-kube-api-access-rpmh8\") pod \"jobset-controller-manager-6d6b4cf5db-c9j8t\" (UID: \"a2d7bf55-7997-4fe9-82a4-192f0c946deb\") " pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"
Oct 30 23:13:09.376575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.376460    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2d7bf55-7997-4fe9-82a4-192f0c946deb-cert\") pod \"jobset-controller-manager-6d6b4cf5db-c9j8t\" (UID: \"a2d7bf55-7997-4fe9-82a4-192f0c946deb\") " pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"
Oct 30 23:13:09.377242 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.377152    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a2d7bf55-7997-4fe9-82a4-192f0c946deb-manager-config\") pod \"jobset-controller-manager-6d6b4cf5db-c9j8t\" (UID: \"a2d7bf55-7997-4fe9-82a4-192f0c946deb\") " pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"
Oct 30 23:13:09.379108 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.379083    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2d7bf55-7997-4fe9-82a4-192f0c946deb-metrics-certs\") pod \"jobset-controller-manager-6d6b4cf5db-c9j8t\" (UID: \"a2d7bf55-7997-4fe9-82a4-192f0c946deb\") " pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"
Oct 30 23:13:09.379108 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.379091    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a2d7bf55-7997-4fe9-82a4-192f0c946deb-cert\") pod \"jobset-controller-manager-6d6b4cf5db-c9j8t\" (UID: \"a2d7bf55-7997-4fe9-82a4-192f0c946deb\") " pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"
Oct 30 23:13:09.438743 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.438681    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpmh8\" (UniqueName: \"kubernetes.io/projected/a2d7bf55-7997-4fe9-82a4-192f0c946deb-kube-api-access-rpmh8\") pod \"jobset-controller-manager-6d6b4cf5db-c9j8t\" (UID: \"a2d7bf55-7997-4fe9-82a4-192f0c946deb\") " pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"
Oct 30 23:13:09.489731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.489707    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"
Oct 30 23:13:09.854975 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:09.854899    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"]
Oct 30 23:13:09.857436 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:13:09.857408    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d7bf55_7997_4fe9_82a4_192f0c946deb.slice/crio-4e1f948f4ea8e03395b6dfe9b192b041d79d5df30ab628beaeaf47d189a8f25d WatchSource:0}: Error finding container 4e1f948f4ea8e03395b6dfe9b192b041d79d5df30ab628beaeaf47d189a8f25d: Status 404 returned error can't find the container with id 4e1f948f4ea8e03395b6dfe9b192b041d79d5df30ab628beaeaf47d189a8f25d
Oct 30 23:13:10.569320 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:10.569283    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t" event={"ID":"a2d7bf55-7997-4fe9-82a4-192f0c946deb","Type":"ContainerStarted","Data":"4e1f948f4ea8e03395b6dfe9b192b041d79d5df30ab628beaeaf47d189a8f25d"}
Oct 30 23:13:14.586841 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:14.586805    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t" event={"ID":"a2d7bf55-7997-4fe9-82a4-192f0c946deb","Type":"ContainerStarted","Data":"6d78114c05107c48998134b909bc56be04f883a6afee08fbfb6a778dc8f3acdd"}
Oct 30 23:13:14.587274 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:14.586920    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"
Oct 30 23:13:14.704918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:14.704862    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t" podStartSLOduration=1.5538426699999999 podStartE2EDuration="5.704846504s" totalImagesPullingTime="4.151003834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:13:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:13:14.703673183 +0000 UTC m=+380.591579535" watchObservedRunningTime="2026-10-30 23:13:14.704846504 +0000 UTC m=+380.592752857"
Oct 30 23:13:17.589119 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:13:17.589084    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:13:25.594879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:13:25.594847    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-6d6b4cf5db-c9j8t"
Oct 30 23:13:47.590326 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:13:47.590284    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:14:17.590373 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:14:17.590332    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:14:29.850290 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:29.850257    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"]
Oct 30 23:14:29.853605 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:29.853571    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:29.956043 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:29.956011    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"]
Oct 30 23:14:29.963038 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:29.963013    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gzwt\" (UniqueName: \"kubernetes.io/projected/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-kube-api-access-6gzwt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:29.963180 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:29.963051    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:29.963180 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:29.963156    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:29.963280 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:29.963201    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:29.963336 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:29.963280    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:29.963336 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:29.963304    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:29.963439 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:29.963382    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:30.064291 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.064243    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:30.064291 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.064290    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:30.064546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.064330    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:30.064546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.064369    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gzwt\" (UniqueName: \"kubernetes.io/projected/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-kube-api-access-6gzwt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:30.064546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.064401    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:30.064546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.064445    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:30.064546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.064471    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:30.065513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.065480    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:30.066897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.066875    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:30.067022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.067003    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:30.067107 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.067091    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:30.067142 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.067100    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:30.132063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.131998    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gzwt\" (UniqueName: \"kubernetes.io/projected/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-kube-api-access-6gzwt\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:30.132180 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.132068    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b80c6f1-0c87-4ae4-9ffc-c24b97741f93-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rvtqr\" (UID: \"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:30.164921 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.164892    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:30.591326 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.591289    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"]
Oct 30 23:14:30.594244 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:14:30.594220    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b80c6f1_0c87_4ae4_9ffc_c24b97741f93.slice/crio-148d03f0e66bf4aaf8ef98294a200ffa33e3d7cc9653a7183ea0f4c4df1f5d06 WatchSource:0}: Error finding container 148d03f0e66bf4aaf8ef98294a200ffa33e3d7cc9653a7183ea0f4c4df1f5d06: Status 404 returned error can't find the container with id 148d03f0e66bf4aaf8ef98294a200ffa33e3d7cc9653a7183ea0f4c4df1f5d06
Oct 30 23:14:30.596284 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.596255    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:14:30.596356 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.596320    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:14:30.846535 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.846448    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr" event={"ID":"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93","Type":"ContainerStarted","Data":"e95ebbe22c2c184e14f728f5310922cdab5e29ebc94ea015172989a8a86c83aa"}
Oct 30 23:14:30.846535 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.846493    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr" event={"ID":"4b80c6f1-0c87-4ae4-9ffc-c24b97741f93","Type":"ContainerStarted","Data":"148d03f0e66bf4aaf8ef98294a200ffa33e3d7cc9653a7183ea0f4c4df1f5d06"}
Oct 30 23:14:30.846535 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:30.846527    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:31.023961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:31.023852    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr" podStartSLOduration=2.023833914 podStartE2EDuration="2.023833914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:14:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:14:31.022345441 +0000 UTC m=+456.910251795" watchObservedRunningTime="2026-10-30 23:14:31.023833914 +0000 UTC m=+456.911740264"
Oct 30 23:14:31.852146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:31.852116    5078 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rvtqr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:14:31.852312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:31.852165    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr" podUID="4b80c6f1-0c87-4ae4-9ffc-c24b97741f93" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:14:34.851784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:34.851700    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rvtqr"
Oct 30 23:14:35.085528 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.085496    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"]
Oct 30 23:14:35.085734 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.085714    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp" podUID="af0356e9-fe3e-4123-9be2-41f76c02c78b" containerName="discovery" containerID="cri-o://f7f447eacdca39bff606b888344b94e3e4c36534185b9df5851916f93d26d4d9" gracePeriod=30
Oct 30 23:14:35.321370 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.321345    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:14:35.406276 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.406244    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-cacerts\" (UniqueName: \"kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-cacerts\") pod \"af0356e9-fe3e-4123-9be2-41f76c02c78b\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") "
Oct 30 23:14:35.406461 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.406324    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-kubeconfig\") pod \"af0356e9-fe3e-4123-9be2-41f76c02c78b\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") "
Oct 30 23:14:35.406461 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.406348    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-csr-ca-configmap\") pod \"af0356e9-fe3e-4123-9be2-41f76c02c78b\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") "
Oct 30 23:14:35.406461 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.406373    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af0356e9-fe3e-4123-9be2-41f76c02c78b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0356e9-fe3e-4123-9be2-41f76c02c78b-local-certs\") pod \"af0356e9-fe3e-4123-9be2-41f76c02c78b\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") "
Oct 30 23:14:35.406461 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.406413    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0356e9-fe3e-4123-9be2-41f76c02c78b-kube-api-access-89grx\" (UniqueName: \"kubernetes.io/projected/af0356e9-fe3e-4123-9be2-41f76c02c78b-kube-api-access-89grx\") pod \"af0356e9-fe3e-4123-9be2-41f76c02c78b\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") "
Oct 30 23:14:35.406461 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.406448    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-token\" (UniqueName: \"kubernetes.io/projected/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-token\") pod \"af0356e9-fe3e-4123-9be2-41f76c02c78b\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") "
Oct 30 23:14:35.406746 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.406473    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-csr-dns-cert\") pod \"af0356e9-fe3e-4123-9be2-41f76c02c78b\" (UID: \"af0356e9-fe3e-4123-9be2-41f76c02c78b\") "
Oct 30 23:14:35.406895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.406843    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-csr-ca-configmap" pod "af0356e9-fe3e-4123-9be2-41f76c02c78b" (UID: "af0356e9-fe3e-4123-9be2-41f76c02c78b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:14:35.409029 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.408993    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0356e9-fe3e-4123-9be2-41f76c02c78b-kube-api-access-89grx" pod "af0356e9-fe3e-4123-9be2-41f76c02c78b" (UID: "af0356e9-fe3e-4123-9be2-41f76c02c78b"). InnerVolumeSpecName "kube-api-access-89grx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:14:35.409029 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.409012    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-token" pod "af0356e9-fe3e-4123-9be2-41f76c02c78b" (UID: "af0356e9-fe3e-4123-9be2-41f76c02c78b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:14:35.409186 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.409124    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-cacerts" pod "af0356e9-fe3e-4123-9be2-41f76c02c78b" (UID: "af0356e9-fe3e-4123-9be2-41f76c02c78b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:14:35.409186 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.409141    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-kubeconfig" pod "af0356e9-fe3e-4123-9be2-41f76c02c78b" (UID: "af0356e9-fe3e-4123-9be2-41f76c02c78b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:14:35.409292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.409273    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-csr-dns-cert" pod "af0356e9-fe3e-4123-9be2-41f76c02c78b" (UID: "af0356e9-fe3e-4123-9be2-41f76c02c78b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:14:35.409719 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.409700    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af0356e9-fe3e-4123-9be2-41f76c02c78b-local-certs" pod "af0356e9-fe3e-4123-9be2-41f76c02c78b" (UID: "af0356e9-fe3e-4123-9be2-41f76c02c78b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:14:35.507573 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.507498    5078 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-kubeconfig\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:14:35.507573 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.507526    5078 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-csr-ca-configmap\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:14:35.507573 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.507538    5078 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/af0356e9-fe3e-4123-9be2-41f76c02c78b-local-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:14:35.507573 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.507546    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89grx\" (UniqueName: \"kubernetes.io/projected/af0356e9-fe3e-4123-9be2-41f76c02c78b-kube-api-access-89grx\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:14:35.507573 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.507554    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:14:35.507573 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.507562    5078 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-istio-csr-dns-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:14:35.507573 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.507570    5078 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/af0356e9-fe3e-4123-9be2-41f76c02c78b-cacerts\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:14:35.866350 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.866320    5078 generic.go:356] "Generic (PLEG): container finished" podID="af0356e9-fe3e-4123-9be2-41f76c02c78b" containerID="f7f447eacdca39bff606b888344b94e3e4c36534185b9df5851916f93d26d4d9" exitCode=0
Oct 30 23:14:35.866813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.866377    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp" event={"ID":"af0356e9-fe3e-4123-9be2-41f76c02c78b","Type":"ContainerDied","Data":"f7f447eacdca39bff606b888344b94e3e4c36534185b9df5851916f93d26d4d9"}
Oct 30 23:14:35.866813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.866413    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp" event={"ID":"af0356e9-fe3e-4123-9be2-41f76c02c78b","Type":"ContainerDied","Data":"c7820797cd06b9ef75bc6c07c022297bcca1c605243bd623a6134b533bed2d21"}
Oct 30 23:14:35.866813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.866431    5078 scope.go:122] "RemoveContainer" containerID="f7f447eacdca39bff606b888344b94e3e4c36534185b9df5851916f93d26d4d9"
Oct 30 23:14:35.866813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.866380    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"
Oct 30 23:14:35.874721 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.874704    5078 scope.go:122] "RemoveContainer" containerID="f7f447eacdca39bff606b888344b94e3e4c36534185b9df5851916f93d26d4d9"
Oct 30 23:14:35.874977 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:14:35.874958    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f447eacdca39bff606b888344b94e3e4c36534185b9df5851916f93d26d4d9\": container with ID starting with f7f447eacdca39bff606b888344b94e3e4c36534185b9df5851916f93d26d4d9 not found: ID does not exist" containerID="f7f447eacdca39bff606b888344b94e3e4c36534185b9df5851916f93d26d4d9"
Oct 30 23:14:35.875045 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:35.874987    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f447eacdca39bff606b888344b94e3e4c36534185b9df5851916f93d26d4d9"} err="failed to get container status \"f7f447eacdca39bff606b888344b94e3e4c36534185b9df5851916f93d26d4d9\": rpc error: code = NotFound desc = could not find container \"f7f447eacdca39bff606b888344b94e3e4c36534185b9df5851916f93d26d4d9\": container with ID starting with f7f447eacdca39bff606b888344b94e3e4c36534185b9df5851916f93d26d4d9 not found: ID does not exist"
Oct 30 23:14:36.055606 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:36.055572    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"]
Oct 30 23:14:36.105386 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:36.105355    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-xz7tp"]
Oct 30 23:14:36.295367 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:14:36.295339    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af0356e9-fe3e-4123-9be2-41f76c02c78b" path="/var/lib/kubelet/pods/af0356e9-fe3e-4123-9be2-41f76c02c78b/volumes"
Oct 30 23:14:47.589134 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:14:47.589098    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:15:17.589866 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:15:17.589827    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:15:47.590200 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:15:47.590098    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:16:17.590062 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:16:17.590026    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:16:47.589124 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:16:47.589083    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:16:59.292528 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.292495    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qwb8z"]
Oct 30 23:16:59.293034 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.292766    5078 cpu_manager.go:414] "containerMap: removing container" podUID="af0356e9-fe3e-4123-9be2-41f76c02c78b" containerName="discovery"
Oct 30 23:16:59.293034 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.292777    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0356e9-fe3e-4123-9be2-41f76c02c78b" containerName="discovery"
Oct 30 23:16:59.293034 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.292824    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0356e9-fe3e-4123-9be2-41f76c02c78b" containerName="discovery"
Oct 30 23:16:59.295575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.295560    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.341735 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.341704    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70cda85-5da1-43d7-b94b-175bbaec28a5-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qwb8z\" (UID: \"f70cda85-5da1-43d7-b94b-175bbaec28a5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.341735 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.341739    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70cda85-5da1-43d7-b94b-175bbaec28a5-home\") pod \"echo-simulator-kserve-64b6f8f96c-qwb8z\" (UID: \"f70cda85-5da1-43d7-b94b-175bbaec28a5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.341913 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.341763    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70cda85-5da1-43d7-b94b-175bbaec28a5-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qwb8z\" (UID: \"f70cda85-5da1-43d7-b94b-175bbaec28a5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.341913 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.341870    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdjgs\" (UniqueName: \"kubernetes.io/projected/f70cda85-5da1-43d7-b94b-175bbaec28a5-kube-api-access-rdjgs\") pod \"echo-simulator-kserve-64b6f8f96c-qwb8z\" (UID: \"f70cda85-5da1-43d7-b94b-175bbaec28a5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.341913 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.341912    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70cda85-5da1-43d7-b94b-175bbaec28a5-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qwb8z\" (UID: \"f70cda85-5da1-43d7-b94b-175bbaec28a5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.342013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.341933    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70cda85-5da1-43d7-b94b-175bbaec28a5-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qwb8z\" (UID: \"f70cda85-5da1-43d7-b94b-175bbaec28a5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.347935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.347905    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 30 23:16:59.398046 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.397998    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fzxnc\""
Oct 30 23:16:59.398046 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.398015    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 30 23:16:59.398295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.398256    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 30 23:16:59.399991 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.399963    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qwb8z"]
Oct 30 23:16:59.442567 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.442531    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70cda85-5da1-43d7-b94b-175bbaec28a5-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qwb8z\" (UID: \"f70cda85-5da1-43d7-b94b-175bbaec28a5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.442770 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.442578    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70cda85-5da1-43d7-b94b-175bbaec28a5-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qwb8z\" (UID: \"f70cda85-5da1-43d7-b94b-175bbaec28a5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.442770 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.442631    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70cda85-5da1-43d7-b94b-175bbaec28a5-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qwb8z\" (UID: \"f70cda85-5da1-43d7-b94b-175bbaec28a5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.442770 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.442654    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70cda85-5da1-43d7-b94b-175bbaec28a5-home\") pod \"echo-simulator-kserve-64b6f8f96c-qwb8z\" (UID: \"f70cda85-5da1-43d7-b94b-175bbaec28a5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.442770 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.442685    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70cda85-5da1-43d7-b94b-175bbaec28a5-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qwb8z\" (UID: \"f70cda85-5da1-43d7-b94b-175bbaec28a5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.442770 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.442725    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdjgs\" (UniqueName: \"kubernetes.io/projected/f70cda85-5da1-43d7-b94b-175bbaec28a5-kube-api-access-rdjgs\") pod \"echo-simulator-kserve-64b6f8f96c-qwb8z\" (UID: \"f70cda85-5da1-43d7-b94b-175bbaec28a5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.443256 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.443145    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f70cda85-5da1-43d7-b94b-175bbaec28a5-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qwb8z\" (UID: \"f70cda85-5da1-43d7-b94b-175bbaec28a5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.443324 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.443254    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f70cda85-5da1-43d7-b94b-175bbaec28a5-home\") pod \"echo-simulator-kserve-64b6f8f96c-qwb8z\" (UID: \"f70cda85-5da1-43d7-b94b-175bbaec28a5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.443396 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.443371    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f70cda85-5da1-43d7-b94b-175bbaec28a5-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qwb8z\" (UID: \"f70cda85-5da1-43d7-b94b-175bbaec28a5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.445280 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.445259    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f70cda85-5da1-43d7-b94b-175bbaec28a5-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qwb8z\" (UID: \"f70cda85-5da1-43d7-b94b-175bbaec28a5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.445369 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.445341    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f70cda85-5da1-43d7-b94b-175bbaec28a5-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qwb8z\" (UID: \"f70cda85-5da1-43d7-b94b-175bbaec28a5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.502491 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.502454    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdjgs\" (UniqueName: \"kubernetes.io/projected/f70cda85-5da1-43d7-b94b-175bbaec28a5-kube-api-access-rdjgs\") pod \"echo-simulator-kserve-64b6f8f96c-qwb8z\" (UID: \"f70cda85-5da1-43d7-b94b-175bbaec28a5\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.606726 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.606625    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:16:59.975432 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:16:59.975351    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qwb8z"]
Oct 30 23:16:59.978205 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:16:59.978178    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf70cda85_5da1_43d7_b94b_175bbaec28a5.slice/crio-b8351409f2087d65d68c1f98fbc6eb7c17b6889a5239e5cb44f5a37d40fa1050 WatchSource:0}: Error finding container b8351409f2087d65d68c1f98fbc6eb7c17b6889a5239e5cb44f5a37d40fa1050: Status 404 returned error can't find the container with id b8351409f2087d65d68c1f98fbc6eb7c17b6889a5239e5cb44f5a37d40fa1050
Oct 30 23:17:00.354352 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:00.354318    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z" event={"ID":"f70cda85-5da1-43d7-b94b-175bbaec28a5","Type":"ContainerStarted","Data":"b8351409f2087d65d68c1f98fbc6eb7c17b6889a5239e5cb44f5a37d40fa1050"}
Oct 30 23:17:03.366420 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:03.366383    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z" event={"ID":"f70cda85-5da1-43d7-b94b-175bbaec28a5","Type":"ContainerStarted","Data":"47e8fbe3e0a1151b52fcc406843d76c695107bbc084f84bc969e69722233943c"}
Oct 30 23:17:03.487356 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:03.487303    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z" podStartSLOduration=1.813802814 podStartE2EDuration="4.487288185s" totalImagesPullingTime="2.673485371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:16:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:17:03.485914696 +0000 UTC m=+609.373821049" watchObservedRunningTime="2026-10-30 23:17:03.487288185 +0000 UTC m=+609.375194537"
Oct 30 23:17:03.607579 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:03.607542    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:17:06.628061 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.628027    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"]
Oct 30 23:17:06.631085 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.631069    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.700507 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.700477    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7160e014-9b36-4fc3-afc6-82aa41241b3d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn\" (UID: \"7160e014-9b36-4fc3-afc6-82aa41241b3d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.700676 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.700521    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qm9m\" (UniqueName: \"kubernetes.io/projected/7160e014-9b36-4fc3-afc6-82aa41241b3d-kube-api-access-4qm9m\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn\" (UID: \"7160e014-9b36-4fc3-afc6-82aa41241b3d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.700720 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.700689    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7160e014-9b36-4fc3-afc6-82aa41241b3d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn\" (UID: \"7160e014-9b36-4fc3-afc6-82aa41241b3d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.700755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.700728    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7160e014-9b36-4fc3-afc6-82aa41241b3d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn\" (UID: \"7160e014-9b36-4fc3-afc6-82aa41241b3d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.700788 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.700768    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7160e014-9b36-4fc3-afc6-82aa41241b3d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn\" (UID: \"7160e014-9b36-4fc3-afc6-82aa41241b3d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.700823 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.700816    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7160e014-9b36-4fc3-afc6-82aa41241b3d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn\" (UID: \"7160e014-9b36-4fc3-afc6-82aa41241b3d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.732334 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.732307    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 30 23:17:06.733225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.733201    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"]
Oct 30 23:17:06.801368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.801335    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7160e014-9b36-4fc3-afc6-82aa41241b3d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn\" (UID: \"7160e014-9b36-4fc3-afc6-82aa41241b3d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.801368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.801365    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7160e014-9b36-4fc3-afc6-82aa41241b3d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn\" (UID: \"7160e014-9b36-4fc3-afc6-82aa41241b3d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.801589 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.801387    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7160e014-9b36-4fc3-afc6-82aa41241b3d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn\" (UID: \"7160e014-9b36-4fc3-afc6-82aa41241b3d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.801589 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.801411    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7160e014-9b36-4fc3-afc6-82aa41241b3d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn\" (UID: \"7160e014-9b36-4fc3-afc6-82aa41241b3d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.801589 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.801442    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7160e014-9b36-4fc3-afc6-82aa41241b3d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn\" (UID: \"7160e014-9b36-4fc3-afc6-82aa41241b3d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.801589 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.801480    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qm9m\" (UniqueName: \"kubernetes.io/projected/7160e014-9b36-4fc3-afc6-82aa41241b3d-kube-api-access-4qm9m\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn\" (UID: \"7160e014-9b36-4fc3-afc6-82aa41241b3d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.801863 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.801842    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7160e014-9b36-4fc3-afc6-82aa41241b3d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn\" (UID: \"7160e014-9b36-4fc3-afc6-82aa41241b3d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.801990 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.801916    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7160e014-9b36-4fc3-afc6-82aa41241b3d-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn\" (UID: \"7160e014-9b36-4fc3-afc6-82aa41241b3d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.801990 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.801953    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7160e014-9b36-4fc3-afc6-82aa41241b3d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn\" (UID: \"7160e014-9b36-4fc3-afc6-82aa41241b3d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.803806 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.803791    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7160e014-9b36-4fc3-afc6-82aa41241b3d-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn\" (UID: \"7160e014-9b36-4fc3-afc6-82aa41241b3d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.804053 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.804037    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7160e014-9b36-4fc3-afc6-82aa41241b3d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn\" (UID: \"7160e014-9b36-4fc3-afc6-82aa41241b3d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.859673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.859643    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qm9m\" (UniqueName: \"kubernetes.io/projected/7160e014-9b36-4fc3-afc6-82aa41241b3d-kube-api-access-4qm9m\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn\" (UID: \"7160e014-9b36-4fc3-afc6-82aa41241b3d\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:06.950104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:06.950028    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:07.319681 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:07.319645    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"]
Oct 30 23:17:07.322684 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:17:07.322655    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7160e014_9b36_4fc3_afc6_82aa41241b3d.slice/crio-9f59b2b50ed2d47cdb1438bf3f0c853ae69a48968b58f5bbd91656eba4f947c4 WatchSource:0}: Error finding container 9f59b2b50ed2d47cdb1438bf3f0c853ae69a48968b58f5bbd91656eba4f947c4: Status 404 returned error can't find the container with id 9f59b2b50ed2d47cdb1438bf3f0c853ae69a48968b58f5bbd91656eba4f947c4
Oct 30 23:17:07.381584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:07.381556    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" event={"ID":"7160e014-9b36-4fc3-afc6-82aa41241b3d","Type":"ContainerStarted","Data":"9f59b2b50ed2d47cdb1438bf3f0c853ae69a48968b58f5bbd91656eba4f947c4"}
Oct 30 23:17:09.391057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:09.391017    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" event={"ID":"7160e014-9b36-4fc3-afc6-82aa41241b3d","Type":"ContainerStarted","Data":"05bb52d9d5a041e65e72f3ea3e7e01d6777832828cbc3584e7b5aea5fdcac044"}
Oct 30 23:17:09.505925 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:09.505864    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podStartSLOduration=2.054093254 podStartE2EDuration="3.505849886s" totalImagesPullingTime="1.451756632s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:17:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:17:09.505762011 +0000 UTC m=+615.393668376" watchObservedRunningTime="2026-10-30 23:17:09.505849886 +0000 UTC m=+615.393756239"
Oct 30 23:17:09.607698 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:09.607657    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:17:09.620931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:09.620898    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:17:09.950892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:09.950862    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:10.409838 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:10.409094    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qwb8z"
Oct 30 23:17:16.951263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:16.951222    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:16.963525 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:16.963499    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:17.144493 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.144461    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"]
Oct 30 23:17:17.147782 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.147767    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.177786 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.177759    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82a135d9-df91-4118-a6fe-feb2d0f4aeba-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j6k6v\" (UID: \"82a135d9-df91-4118-a6fe-feb2d0f4aeba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.177926 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.177790    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82a135d9-df91-4118-a6fe-feb2d0f4aeba-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j6k6v\" (UID: \"82a135d9-df91-4118-a6fe-feb2d0f4aeba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.177926 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.177811    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82a135d9-df91-4118-a6fe-feb2d0f4aeba-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j6k6v\" (UID: \"82a135d9-df91-4118-a6fe-feb2d0f4aeba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.178056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.177922    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a135d9-df91-4118-a6fe-feb2d0f4aeba-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j6k6v\" (UID: \"82a135d9-df91-4118-a6fe-feb2d0f4aeba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.178056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.178029    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82a135d9-df91-4118-a6fe-feb2d0f4aeba-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j6k6v\" (UID: \"82a135d9-df91-4118-a6fe-feb2d0f4aeba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.178133 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.178079    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6hcq\" (UniqueName: \"kubernetes.io/projected/82a135d9-df91-4118-a6fe-feb2d0f4aeba-kube-api-access-f6hcq\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j6k6v\" (UID: \"82a135d9-df91-4118-a6fe-feb2d0f4aeba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.200128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.200104    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 30 23:17:17.206842 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.206791    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"]
Oct 30 23:17:17.278686 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.278664    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a135d9-df91-4118-a6fe-feb2d0f4aeba-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j6k6v\" (UID: \"82a135d9-df91-4118-a6fe-feb2d0f4aeba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.278839 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.278702    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82a135d9-df91-4118-a6fe-feb2d0f4aeba-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j6k6v\" (UID: \"82a135d9-df91-4118-a6fe-feb2d0f4aeba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.278839 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.278724    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6hcq\" (UniqueName: \"kubernetes.io/projected/82a135d9-df91-4118-a6fe-feb2d0f4aeba-kube-api-access-f6hcq\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j6k6v\" (UID: \"82a135d9-df91-4118-a6fe-feb2d0f4aeba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.278954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.278910    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82a135d9-df91-4118-a6fe-feb2d0f4aeba-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j6k6v\" (UID: \"82a135d9-df91-4118-a6fe-feb2d0f4aeba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.278954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.278947    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82a135d9-df91-4118-a6fe-feb2d0f4aeba-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j6k6v\" (UID: \"82a135d9-df91-4118-a6fe-feb2d0f4aeba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.279063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.278974    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82a135d9-df91-4118-a6fe-feb2d0f4aeba-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j6k6v\" (UID: \"82a135d9-df91-4118-a6fe-feb2d0f4aeba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.279179 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.279159    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/82a135d9-df91-4118-a6fe-feb2d0f4aeba-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j6k6v\" (UID: \"82a135d9-df91-4118-a6fe-feb2d0f4aeba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.279234 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.279179    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a135d9-df91-4118-a6fe-feb2d0f4aeba-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j6k6v\" (UID: \"82a135d9-df91-4118-a6fe-feb2d0f4aeba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.279297 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.279281    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/82a135d9-df91-4118-a6fe-feb2d0f4aeba-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j6k6v\" (UID: \"82a135d9-df91-4118-a6fe-feb2d0f4aeba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.281442 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.281417    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/82a135d9-df91-4118-a6fe-feb2d0f4aeba-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j6k6v\" (UID: \"82a135d9-df91-4118-a6fe-feb2d0f4aeba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.281585 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.281566    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/82a135d9-df91-4118-a6fe-feb2d0f4aeba-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j6k6v\" (UID: \"82a135d9-df91-4118-a6fe-feb2d0f4aeba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.335722 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.335695    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6hcq\" (UniqueName: \"kubernetes.io/projected/82a135d9-df91-4118-a6fe-feb2d0f4aeba-kube-api-access-f6hcq\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-j6k6v\" (UID: \"82a135d9-df91-4118-a6fe-feb2d0f4aeba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.429787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.429762    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:17:17.459093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.459036    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:17.592802 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:17:17.592771    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:17:17.828261 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:17.828232    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"]
Oct 30 23:17:17.830934 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:17:17.830911    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a135d9_df91_4118_a6fe_feb2d0f4aeba.slice/crio-0ba56efd6850a0a2542ddd5760d063f8a1286e89ba15be0f28e77ad67ab34210 WatchSource:0}: Error finding container 0ba56efd6850a0a2542ddd5760d063f8a1286e89ba15be0f28e77ad67ab34210: Status 404 returned error can't find the container with id 0ba56efd6850a0a2542ddd5760d063f8a1286e89ba15be0f28e77ad67ab34210
Oct 30 23:17:18.424085 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:18.424062    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v" event={"ID":"82a135d9-df91-4118-a6fe-feb2d0f4aeba","Type":"ContainerStarted","Data":"0ba56efd6850a0a2542ddd5760d063f8a1286e89ba15be0f28e77ad67ab34210"}
Oct 30 23:17:19.428971 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:19.428940    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v" event={"ID":"82a135d9-df91-4118-a6fe-feb2d0f4aeba","Type":"ContainerStarted","Data":"e1cbe3655b36f6414eb632802192fe71cb4657c4f433cfc78f9dd76c3f3a121b"}
Oct 30 23:17:19.459687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:19.459662    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:19.553909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:19.553864    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v" podStartSLOduration=1.96476752 podStartE2EDuration="2.553846947s" totalImagesPullingTime="589.079427ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:17:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:17:19.552305045 +0000 UTC m=+625.440211398" watchObservedRunningTime="2026-10-30 23:17:19.553846947 +0000 UTC m=+625.441753300"
Oct 30 23:17:22.326796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.326768    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"]
Oct 30 23:17:22.329978 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.329963    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.381773 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.381748    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-l8k5z\""
Oct 30 23:17:22.417918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.417887    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.418103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.417949    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.418103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.417992    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.418189 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.418126    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.418224 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.418210    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-688mx\" (UniqueName: \"kubernetes.io/projected/46d4c9f2-e2ca-4f73-8edb-e532fc479059-kube-api-access-688mx\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.418282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.418262    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.418374 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.418352    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.418469 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.418457    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.418524 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.418485    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.432906 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.432883    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"]
Oct 30 23:17:22.518978 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.518948    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.518978 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.518980    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.519176 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.519008    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.519176 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.519128    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.519176 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.519172    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.519334 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.519205    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.519334 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.519246    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-688mx\" (UniqueName: \"kubernetes.io/projected/46d4c9f2-e2ca-4f73-8edb-e532fc479059-kube-api-access-688mx\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.519334 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.519292    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.519334 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.519313    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.519525 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.519414    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.519525 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.519447    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.519660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.519639    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.519812 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.519786    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.519933 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.519872    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.521589 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.521568    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.521875 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.521859    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.579015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.578950    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.579125 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.579058    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-688mx\" (UniqueName: \"kubernetes.io/projected/46d4c9f2-e2ca-4f73-8edb-e532fc479059-kube-api-access-688mx\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q4jfd\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:22.642148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:22.642122    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:23.064002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:23.063825    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"]
Oct 30 23:17:23.066382 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:17:23.066357    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d4c9f2_e2ca_4f73_8edb_e532fc479059.slice/crio-9234fdc1d5e8a007f8dff4b8ac11114d388bb6607eea63850edd625768bbf8cc WatchSource:0}: Error finding container 9234fdc1d5e8a007f8dff4b8ac11114d388bb6607eea63850edd625768bbf8cc: Status 404 returned error can't find the container with id 9234fdc1d5e8a007f8dff4b8ac11114d388bb6607eea63850edd625768bbf8cc
Oct 30 23:17:23.446716 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:23.446639    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd" event={"ID":"46d4c9f2-e2ca-4f73-8edb-e532fc479059","Type":"ContainerStarted","Data":"9234fdc1d5e8a007f8dff4b8ac11114d388bb6607eea63850edd625768bbf8cc"}
Oct 30 23:17:27.459340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:27.459298    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:27.461872 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:27.461844    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:17:27.461980 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:27.461906    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:17:27.461980 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:27.461936    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:17:27.472236 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:27.472220    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:28.483627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:28.479354    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-j6k6v"
Oct 30 23:17:28.486921 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:28.486883    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd" event={"ID":"46d4c9f2-e2ca-4f73-8edb-e532fc479059","Type":"ContainerStarted","Data":"86e1a50c943c7216ba56615f8beca4e2f562dc34efcbbd163f14bd26b321882e"}
Oct 30 23:17:28.642961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:28.642930    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:28.644165 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:28.644140    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q4jfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Oct 30 23:17:28.644312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:28.644185    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd" podUID="46d4c9f2-e2ca-4f73-8edb-e532fc479059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Oct 30 23:17:28.714459 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:28.714405    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd" podStartSLOduration=2.321667427 podStartE2EDuration="6.714388073s" totalImagesPullingTime="4.392720646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:17:28.712749263 +0000 UTC m=+634.600655620" watchObservedRunningTime="2026-10-30 23:17:28.714388073 +0000 UTC m=+634.602294425"
Oct 30 23:17:29.643042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:29.643008    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q4jfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Oct 30 23:17:29.643453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:29.643061    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd" podUID="46d4c9f2-e2ca-4f73-8edb-e532fc479059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Oct 30 23:17:30.643291 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:30.643254    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q4jfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Oct 30 23:17:30.643882 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:30.643319    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd" podUID="46d4c9f2-e2ca-4f73-8edb-e532fc479059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Oct 30 23:17:31.643451 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:31.643417    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q4jfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Oct 30 23:17:31.643882 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:31.643480    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd" podUID="46d4c9f2-e2ca-4f73-8edb-e532fc479059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Oct 30 23:17:32.643075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:32.643044    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q4jfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Oct 30 23:17:32.643247 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:32.643103    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd" podUID="46d4c9f2-e2ca-4f73-8edb-e532fc479059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Oct 30 23:17:33.643089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:33.643008    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q4jfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Oct 30 23:17:33.643089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:33.643058    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd" podUID="46d4c9f2-e2ca-4f73-8edb-e532fc479059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Oct 30 23:17:34.643127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:34.643088    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q4jfd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body=
Oct 30 23:17:34.643534 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:34.643148    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd" podUID="46d4c9f2-e2ca-4f73-8edb-e532fc479059" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused"
Oct 30 23:17:35.646873 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:35.646847    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:35.647342 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:35.647126    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:35.648242 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:35.648223    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:17:36.765148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:36.765105    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z"]
Oct 30 23:17:36.769916 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:36.769891    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z"
Oct 30 23:17:36.854175 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:36.854146    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 23:17:36.854396 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:36.854262    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 30 23:17:36.854396 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:36.854308    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-jckcl\""
Oct 30 23:17:36.854956 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:36.854509    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n7d8\" (UniqueName: \"kubernetes.io/projected/ae733494-5294-4fa5-a378-bc9214f8a5a8-kube-api-access-9n7d8\") pod \"ai-gateway-controller-5d9cd99f45-bfg6z\" (UID: \"ae733494-5294-4fa5-a378-bc9214f8a5a8\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z"
Oct 30 23:17:36.854956 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:36.854578    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae733494-5294-4fa5-a378-bc9214f8a5a8-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-bfg6z\" (UID: \"ae733494-5294-4fa5-a378-bc9214f8a5a8\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z"
Oct 30 23:17:36.873399 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:36.873369    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 30 23:17:36.873552 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:36.873406    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 23:17:36.874483 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:36.874455    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z"]
Oct 30 23:17:36.955306 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:36.955274    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n7d8\" (UniqueName: \"kubernetes.io/projected/ae733494-5294-4fa5-a378-bc9214f8a5a8-kube-api-access-9n7d8\") pod \"ai-gateway-controller-5d9cd99f45-bfg6z\" (UID: \"ae733494-5294-4fa5-a378-bc9214f8a5a8\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z"
Oct 30 23:17:36.955452 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:36.955321    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae733494-5294-4fa5-a378-bc9214f8a5a8-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-bfg6z\" (UID: \"ae733494-5294-4fa5-a378-bc9214f8a5a8\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z"
Oct 30 23:17:36.957854 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:36.957836    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae733494-5294-4fa5-a378-bc9214f8a5a8-metrics-certs\") pod \"ai-gateway-controller-5d9cd99f45-bfg6z\" (UID: \"ae733494-5294-4fa5-a378-bc9214f8a5a8\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z"
Oct 30 23:17:37.020643 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.020568    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n7d8\" (UniqueName: \"kubernetes.io/projected/ae733494-5294-4fa5-a378-bc9214f8a5a8-kube-api-access-9n7d8\") pod \"ai-gateway-controller-5d9cd99f45-bfg6z\" (UID: \"ae733494-5294-4fa5-a378-bc9214f8a5a8\") " pod="opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z"
Oct 30 23:17:37.082688 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.082664    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z"
Oct 30 23:17:37.458394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.458368    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z"]
Oct 30 23:17:37.461574 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:17:37.461545    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae733494_5294_4fa5_a378_bc9214f8a5a8.slice/crio-deadea7ffca426e376dae96d3631a404014ee1d8d7d42a6d45449f65bed2b5ad WatchSource:0}: Error finding container deadea7ffca426e376dae96d3631a404014ee1d8d7d42a6d45449f65bed2b5ad: Status 404 returned error can't find the container with id deadea7ffca426e376dae96d3631a404014ee1d8d7d42a6d45449f65bed2b5ad
Oct 30 23:17:37.463092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.463074    5078 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:17:37.468606 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.468571    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"]
Oct 30 23:17:37.473612 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.473577    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.521813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.521782    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z" event={"ID":"ae733494-5294-4fa5-a378-bc9214f8a5a8","Type":"ContainerStarted","Data":"deadea7ffca426e376dae96d3631a404014ee1d8d7d42a6d45449f65bed2b5ad"}
Oct 30 23:17:37.560952 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.560925    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.561094 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.560962    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.561094 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.560980    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.561094 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.561072    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.561237 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.561113    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.561237 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.561146    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w9pz\" (UniqueName: \"kubernetes.io/projected/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-kube-api-access-9w9pz\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.561237 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.561168    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.561347 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.561242    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.561347 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.561264    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.570000 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.569973    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"]
Oct 30 23:17:37.661785 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.661759    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.661954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.661802    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.662020 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.661983    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.662078 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.662053    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.662134 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.662101    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.662193 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.662141    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.662193 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.662168    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.662292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.662194    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w9pz\" (UniqueName: \"kubernetes.io/projected/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-kube-api-access-9w9pz\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.662292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.662239    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.662409 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.662387    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.662475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.662419    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.662531 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.662508    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.662689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.662670    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.662761 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.662687    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.664913 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.664884    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.665106 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.665090    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.719262 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.719205    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.719341 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.719318    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w9pz\" (UniqueName: \"kubernetes.io/projected/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-kube-api-access-9w9pz\") pod \"maas-default-gateway-openshift-default-5df48fcc99-rwc77\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:37.787115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:37.787092    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:38.208987 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:38.208926    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"]
Oct 30 23:17:38.211329 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:17:38.211298    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf434e905_62f0_4aef_acf5_1ba6f49ff6f0.slice/crio-43598cf89c2e6da6fd00a87e53463d7851df2af406b6b7dfd8d2ef23761c73a9 WatchSource:0}: Error finding container 43598cf89c2e6da6fd00a87e53463d7851df2af406b6b7dfd8d2ef23761c73a9: Status 404 returned error can't find the container with id 43598cf89c2e6da6fd00a87e53463d7851df2af406b6b7dfd8d2ef23761c73a9
Oct 30 23:17:38.214432 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:38.214404    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:17:38.214519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:38.214463    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:17:38.214519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:38.214491    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:17:38.527982 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:38.527904    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77" event={"ID":"f434e905-62f0-4aef-acf5-1ba6f49ff6f0","Type":"ContainerStarted","Data":"6e23bff946b8ed3e5b31462ce956fe5fca3e025f09dc16829c70672490fe86a4"}
Oct 30 23:17:38.527982 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:38.527959    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77" event={"ID":"f434e905-62f0-4aef-acf5-1ba6f49ff6f0","Type":"ContainerStarted","Data":"43598cf89c2e6da6fd00a87e53463d7851df2af406b6b7dfd8d2ef23761c73a9"}
Oct 30 23:17:38.644821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:38.644738    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77" podStartSLOduration=1.644717946 podStartE2EDuration="1.644717946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:17:38.643429559 +0000 UTC m=+644.531335909" watchObservedRunningTime="2026-10-30 23:17:38.644717946 +0000 UTC m=+644.532624300"
Oct 30 23:17:38.788678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:38.788168    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:39.789849 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:39.789813    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rwc77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Oct 30 23:17:39.790262 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:39.789883    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77" podUID="f434e905-62f0-4aef-acf5-1ba6f49ff6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Oct 30 23:17:40.788162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:40.788128    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rwc77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Oct 30 23:17:40.788348 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:40.788189    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77" podUID="f434e905-62f0-4aef-acf5-1ba6f49ff6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Oct 30 23:17:41.541183 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:41.541143    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z" event={"ID":"ae733494-5294-4fa5-a378-bc9214f8a5a8","Type":"ContainerStarted","Data":"a95fa93a75c3f9d6b2295efe4fef2bda1a742a1bfda05627762bb6bab29ef886"}
Oct 30 23:17:41.541573 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:41.541262    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z"
Oct 30 23:17:41.656860 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:41.656801    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z" podStartSLOduration=1.97288951 podStartE2EDuration="5.656784189s" totalImagesPullingTime="3.683894679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:17:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:17:41.656457857 +0000 UTC m=+647.544364243" watchObservedRunningTime="2026-10-30 23:17:41.656784189 +0000 UTC m=+647.544690545"
Oct 30 23:17:41.788295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:41.788253    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rwc77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Oct 30 23:17:41.788505 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:41.788322    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77" podUID="f434e905-62f0-4aef-acf5-1ba6f49ff6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Oct 30 23:17:42.788480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:42.788444    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rwc77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Oct 30 23:17:42.788931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:42.788505    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77" podUID="f434e905-62f0-4aef-acf5-1ba6f49ff6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Oct 30 23:17:43.788260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:43.788222    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rwc77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Oct 30 23:17:43.788433 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:43.788291    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77" podUID="f434e905-62f0-4aef-acf5-1ba6f49ff6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Oct 30 23:17:44.787920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:44.787880    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rwc77 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused" start-of-body=
Oct 30 23:17:44.788307 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:44.787961    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77" podUID="f434e905-62f0-4aef-acf5-1ba6f49ff6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": dial tcp 10.133.0.25:15021: connect: connection refused"
Oct 30 23:17:46.604011 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:46.603978    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:46.604407 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:46.604347    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:46.605208 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:46.605191    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:17:47.596733 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:17:47.596693    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:17:52.548010 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:17:52.547979    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z"
Oct 30 23:18:07.629734 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:07.629698    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z"]
Oct 30 23:18:07.630316 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:07.629979    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z" podUID="ae733494-5294-4fa5-a378-bc9214f8a5a8" containerName="manager" containerID="cri-o://a95fa93a75c3f9d6b2295efe4fef2bda1a742a1bfda05627762bb6bab29ef886" gracePeriod=30
Oct 30 23:18:07.868842 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:07.868815    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z"
Oct 30 23:18:07.937216 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:07.937146    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae733494-5294-4fa5-a378-bc9214f8a5a8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae733494-5294-4fa5-a378-bc9214f8a5a8-metrics-certs\") pod \"ae733494-5294-4fa5-a378-bc9214f8a5a8\" (UID: \"ae733494-5294-4fa5-a378-bc9214f8a5a8\") "
Oct 30 23:18:07.937216 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:07.937191    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae733494-5294-4fa5-a378-bc9214f8a5a8-kube-api-access-9n7d8\" (UniqueName: \"kubernetes.io/projected/ae733494-5294-4fa5-a378-bc9214f8a5a8-kube-api-access-9n7d8\") pod \"ae733494-5294-4fa5-a378-bc9214f8a5a8\" (UID: \"ae733494-5294-4fa5-a378-bc9214f8a5a8\") "
Oct 30 23:18:07.939390 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:07.939365    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae733494-5294-4fa5-a378-bc9214f8a5a8-metrics-certs" pod "ae733494-5294-4fa5-a378-bc9214f8a5a8" (UID: "ae733494-5294-4fa5-a378-bc9214f8a5a8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:18:07.939493 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:07.939463    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae733494-5294-4fa5-a378-bc9214f8a5a8-kube-api-access-9n7d8" pod "ae733494-5294-4fa5-a378-bc9214f8a5a8" (UID: "ae733494-5294-4fa5-a378-bc9214f8a5a8"). InnerVolumeSpecName "kube-api-access-9n7d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:18:08.037947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:08.037919    5078 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae733494-5294-4fa5-a378-bc9214f8a5a8-metrics-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:18:08.037947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:08.037946    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n7d8\" (UniqueName: \"kubernetes.io/projected/ae733494-5294-4fa5-a378-bc9214f8a5a8-kube-api-access-9n7d8\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:18:08.651340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:08.651248    5078 generic.go:356] "Generic (PLEG): container finished" podID="ae733494-5294-4fa5-a378-bc9214f8a5a8" containerID="a95fa93a75c3f9d6b2295efe4fef2bda1a742a1bfda05627762bb6bab29ef886" exitCode=0
Oct 30 23:18:08.651340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:08.651289    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z" event={"ID":"ae733494-5294-4fa5-a378-bc9214f8a5a8","Type":"ContainerDied","Data":"a95fa93a75c3f9d6b2295efe4fef2bda1a742a1bfda05627762bb6bab29ef886"}
Oct 30 23:18:08.651340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:08.651291    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z"
Oct 30 23:18:08.651340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:08.651326    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z" event={"ID":"ae733494-5294-4fa5-a378-bc9214f8a5a8","Type":"ContainerDied","Data":"deadea7ffca426e376dae96d3631a404014ee1d8d7d42a6d45449f65bed2b5ad"}
Oct 30 23:18:08.651952 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:08.651346    5078 scope.go:122] "RemoveContainer" containerID="a95fa93a75c3f9d6b2295efe4fef2bda1a742a1bfda05627762bb6bab29ef886"
Oct 30 23:18:08.659479 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:08.659455    5078 scope.go:122] "RemoveContainer" containerID="a95fa93a75c3f9d6b2295efe4fef2bda1a742a1bfda05627762bb6bab29ef886"
Oct 30 23:18:08.659766 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:18:08.659744    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a95fa93a75c3f9d6b2295efe4fef2bda1a742a1bfda05627762bb6bab29ef886\": container with ID starting with a95fa93a75c3f9d6b2295efe4fef2bda1a742a1bfda05627762bb6bab29ef886 not found: ID does not exist" containerID="a95fa93a75c3f9d6b2295efe4fef2bda1a742a1bfda05627762bb6bab29ef886"
Oct 30 23:18:08.659841 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:08.659779    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a95fa93a75c3f9d6b2295efe4fef2bda1a742a1bfda05627762bb6bab29ef886"} err="failed to get container status \"a95fa93a75c3f9d6b2295efe4fef2bda1a742a1bfda05627762bb6bab29ef886\": rpc error: code = NotFound desc = could not find container \"a95fa93a75c3f9d6b2295efe4fef2bda1a742a1bfda05627762bb6bab29ef886\": container with ID starting with a95fa93a75c3f9d6b2295efe4fef2bda1a742a1bfda05627762bb6bab29ef886 not found: ID does not exist"
Oct 30 23:18:08.814292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:08.814263    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z"]
Oct 30 23:18:08.863909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:08.863878    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d9cd99f45-bfg6z"]
Oct 30 23:18:10.295974 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:10.295940    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae733494-5294-4fa5-a378-bc9214f8a5a8" path="/var/lib/kubelet/pods/ae733494-5294-4fa5-a378-bc9214f8a5a8/volumes"
Oct 30 23:18:17.589932 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:18:17.589892    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:18:38.573958 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.573917    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"]
Oct 30 23:18:38.574401 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.574237    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ae733494-5294-4fa5-a378-bc9214f8a5a8" containerName="manager"
Oct 30 23:18:38.574401 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.574248    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae733494-5294-4fa5-a378-bc9214f8a5a8" containerName="manager"
Oct 30 23:18:38.574401 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.574308    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae733494-5294-4fa5-a378-bc9214f8a5a8" containerName="manager"
Oct 30 23:18:38.577184 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.577156    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.586359 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.586339    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-istio-envoy\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.586457 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.586370    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-credential-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.586457 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.586411    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-workload-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.586457 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.586439    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbde5115-111a-42ac-a735-8be11caf2de6-istio-token\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.586562 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.586532    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbde5115-111a-42ac-a735-8be11caf2de6-istiod-ca-cert\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.586618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.586569    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-istio-data\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.586618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.586589    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf796\" (UniqueName: \"kubernetes.io/projected/dbde5115-111a-42ac-a735-8be11caf2de6-kube-api-access-qf796\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.586686 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.586628    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-workload-certs\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.586716 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.586699    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbde5115-111a-42ac-a735-8be11caf2de6-istio-podinfo\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.657207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.657172    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-5dc3c0-openshift-default-dockercfg-f5r9h\""
Oct 30 23:18:38.680672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.680635    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"]
Oct 30 23:18:38.687069 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.687040    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-istio-envoy\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.687223 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.687076    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-credential-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.687223 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.687108    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-workload-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.687359 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.687252    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbde5115-111a-42ac-a735-8be11caf2de6-istio-token\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.687359 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.687320    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbde5115-111a-42ac-a735-8be11caf2de6-istiod-ca-cert\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.687477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.687374    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-istio-data\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.687477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.687409    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf796\" (UniqueName: \"kubernetes.io/projected/dbde5115-111a-42ac-a735-8be11caf2de6-kube-api-access-qf796\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.687477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.687438    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-workload-certs\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.687477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.687467    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbde5115-111a-42ac-a735-8be11caf2de6-istio-podinfo\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.687477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.687471    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-workload-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.687758 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.687571    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-credential-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.687823 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.687805    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-istio-data\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.687895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.687864    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-workload-certs\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.688094 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.688065    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbde5115-111a-42ac-a735-8be11caf2de6-istiod-ca-cert\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.689661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.689635    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-istio-envoy\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.689986 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.689966    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbde5115-111a-42ac-a735-8be11caf2de6-istio-podinfo\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.745202 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.745162    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbde5115-111a-42ac-a735-8be11caf2de6-istio-token\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.745429 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.745372    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf796\" (UniqueName: \"kubernetes.io/projected/dbde5115-111a-42ac-a735-8be11caf2de6-kube-api-access-qf796\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.886457 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.886381    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"]
Oct 30 23:18:38.889124 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.889102    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:38.890878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.890863    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:38.943654 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.943586    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-aadaa6-openshift-default-dockercfg-6bsq5\""
Oct 30 23:18:38.952918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.952892    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"]
Oct 30 23:18:38.990551 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.990523    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-workload-certs\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:38.990695 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.990631    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-token\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:38.990695 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.990674    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3680396f-1877-4e2e-8226-3591fb4e00e2-istiod-ca-cert\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:38.990695 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.990690    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-workload-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:38.990801 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.990711    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-credential-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:38.990801 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.990771    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-data\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:38.990801 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.990791    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5r9c\" (UniqueName: \"kubernetes.io/projected/3680396f-1877-4e2e-8226-3591fb4e00e2-kube-api-access-h5r9c\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:38.990888 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.990847    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-podinfo\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:38.990946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:38.990933    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-envoy\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.091780 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.091750    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-workload-certs\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.091780 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.091787    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-token\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.092032 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.091816    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3680396f-1877-4e2e-8226-3591fb4e00e2-istiod-ca-cert\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.092032 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.091832    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-workload-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.092032 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.091861    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-credential-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.092032 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.091894    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-data\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.092032 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.091926    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5r9c\" (UniqueName: \"kubernetes.io/projected/3680396f-1877-4e2e-8226-3591fb4e00e2-kube-api-access-h5r9c\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.092032 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.091944    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-podinfo\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.092335 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.092036    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-envoy\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.092335 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.092172    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-workload-certs\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.092335 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.092246    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-workload-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.092427 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.092382    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-data\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.092463 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.092423    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-credential-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.092640 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.092619    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3680396f-1877-4e2e-8226-3591fb4e00e2-istiod-ca-cert\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.094541 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.094514    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-envoy\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.094811 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.094795    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-podinfo\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.148937 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.148880    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5r9c\" (UniqueName: \"kubernetes.io/projected/3680396f-1877-4e2e-8226-3591fb4e00e2-kube-api-access-h5r9c\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.148937 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.148880    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-token\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.212003 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.211969    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:39.320829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.320808    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"]
Oct 30 23:18:39.323132 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:18:39.323105    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbde5115_111a_42ac_a735_8be11caf2de6.slice/crio-4ae9043c8ca15498a7806074dd04234e70d73a634ae31041cc9a113b2bced86b WatchSource:0}: Error finding container 4ae9043c8ca15498a7806074dd04234e70d73a634ae31041cc9a113b2bced86b: Status 404 returned error can't find the container with id 4ae9043c8ca15498a7806074dd04234e70d73a634ae31041cc9a113b2bced86b
Oct 30 23:18:39.325263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.325236    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:18:39.325329 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.325308    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:18:39.325366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.325348    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:18:39.596660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.596626    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"]
Oct 30 23:18:39.605854 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:18:39.605817    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3680396f_1877_4e2e_8226_3591fb4e00e2.slice/crio-3ad69bc151f5c4bd556599d06e263022c160652b8b05b9a71239e261c9c23e8f WatchSource:0}: Error finding container 3ad69bc151f5c4bd556599d06e263022c160652b8b05b9a71239e261c9c23e8f: Status 404 returned error can't find the container with id 3ad69bc151f5c4bd556599d06e263022c160652b8b05b9a71239e261c9c23e8f
Oct 30 23:18:39.608579 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.608548    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:18:39.608693 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.608645    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:18:39.608693 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.608685    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:18:39.771569 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.771531    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr" event={"ID":"dbde5115-111a-42ac-a735-8be11caf2de6","Type":"ContainerStarted","Data":"fa2fff8d7432e3ecb51927a8ca1dfac7ea006c47a0069719253ee440e02fd8ea"}
Oct 30 23:18:39.771786 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.771657    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr" event={"ID":"dbde5115-111a-42ac-a735-8be11caf2de6","Type":"ContainerStarted","Data":"4ae9043c8ca15498a7806074dd04234e70d73a634ae31041cc9a113b2bced86b"}
Oct 30 23:18:39.773253 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.773224    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf" event={"ID":"3680396f-1877-4e2e-8226-3591fb4e00e2","Type":"ContainerStarted","Data":"8e406d786f20b88e30e5f0d6d9a435d1f79a12c927d867b42f4c8c3347659db4"}
Oct 30 23:18:39.773381 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.773277    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf" event={"ID":"3680396f-1877-4e2e-8226-3591fb4e00e2","Type":"ContainerStarted","Data":"3ad69bc151f5c4bd556599d06e263022c160652b8b05b9a71239e261c9c23e8f"}
Oct 30 23:18:39.842082 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.842055    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"]
Oct 30 23:18:39.845325 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.845305    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.889664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.889639    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:39.892038 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.891988    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr" podStartSLOduration=1.891971819 podStartE2EDuration="1.891971819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:39.891076882 +0000 UTC m=+705.778983249" watchObservedRunningTime="2026-10-30 23:18:39.891971819 +0000 UTC m=+705.779878173"
Oct 30 23:18:39.896861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.896825    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-istio-envoy\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.896951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.896867    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7136a40b-0373-4850-81c8-1345ff2b8996-istio-podinfo\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.896951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.896901    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7136a40b-0373-4850-81c8-1345ff2b8996-istio-token\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.897068 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.896956    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-workload-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.897068 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.897000    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-istio-data\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.897068 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.897038    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22fhn\" (UniqueName: \"kubernetes.io/projected/7136a40b-0373-4850-81c8-1345ff2b8996-kube-api-access-22fhn\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.897214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.897074    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-credential-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.897214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.897098    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-workload-certs\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.897214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.897139    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7136a40b-0373-4850-81c8-1345ff2b8996-istiod-ca-cert\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.900662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.900640    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-3c9613-openshift-default-dockercfg-n6hzr\""
Oct 30 23:18:39.906091 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.905777    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"]
Oct 30 23:18:39.998041 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.998010    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-workload-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.998198 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.998050    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-istio-data\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.998198 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.998152    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22fhn\" (UniqueName: \"kubernetes.io/projected/7136a40b-0373-4850-81c8-1345ff2b8996-kube-api-access-22fhn\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.998268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.998194    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-credential-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.998268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.998217    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-workload-certs\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.998268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.998238    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7136a40b-0373-4850-81c8-1345ff2b8996-istiod-ca-cert\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.998362 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.998305    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-istio-envoy\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.998362 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.998329    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7136a40b-0373-4850-81c8-1345ff2b8996-istio-podinfo\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.998423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.998365    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7136a40b-0373-4850-81c8-1345ff2b8996-istio-token\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.998423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.998382    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-workload-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.998485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.998426    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-istio-data\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.998710 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.998683    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-workload-certs\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.998956 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.998934    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-credential-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:39.999243 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:39.999215    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7136a40b-0373-4850-81c8-1345ff2b8996-istiod-ca-cert\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:40.000687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:40.000667    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-istio-envoy\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:40.001173 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:40.001155    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7136a40b-0373-4850-81c8-1345ff2b8996-istio-podinfo\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:40.012539 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:40.012472    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf" podStartSLOduration=2.012456081 podStartE2EDuration="2.012456081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:40.012145845 +0000 UTC m=+705.900052210" watchObservedRunningTime="2026-10-30 23:18:40.012456081 +0000 UTC m=+705.900362434"
Oct 30 23:18:40.055426 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:40.055398    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7136a40b-0373-4850-81c8-1345ff2b8996-istio-token\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:40.055657 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:40.055638    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22fhn\" (UniqueName: \"kubernetes.io/projected/7136a40b-0373-4850-81c8-1345ff2b8996-kube-api-access-22fhn\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:40.159792 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:40.159708    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:40.213619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:40.213233    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:40.225520 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:40.225299    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:40.553565 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:40.553387    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"]
Oct 30 23:18:40.556124 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:18:40.556098    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7136a40b_0373_4850_81c8_1345ff2b8996.slice/crio-177e42a7043fa2ffc51b42a02fcfa157be25934f88c87a27c5491cdc896790b5 WatchSource:0}: Error finding container 177e42a7043fa2ffc51b42a02fcfa157be25934f88c87a27c5491cdc896790b5: Status 404 returned error can't find the container with id 177e42a7043fa2ffc51b42a02fcfa157be25934f88c87a27c5491cdc896790b5
Oct 30 23:18:40.558340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:40.558310    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:18:40.558428 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:40.558370    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:18:40.558428 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:40.558397    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:18:40.779936 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:40.779884    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j" event={"ID":"7136a40b-0373-4850-81c8-1345ff2b8996","Type":"ContainerStarted","Data":"1aa37ceb910cc0810fffe763e4a8a06bc4a6f437b92e79d2899ffd76445d96fb"}
Oct 30 23:18:40.780418 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:40.779950    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j" event={"ID":"7136a40b-0373-4850-81c8-1345ff2b8996","Type":"ContainerStarted","Data":"177e42a7043fa2ffc51b42a02fcfa157be25934f88c87a27c5491cdc896790b5"}
Oct 30 23:18:40.780418 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:40.780384    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:40.781334 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:40.781309    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:18:40.894848 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:40.894778    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:40.899226 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:40.899183    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j" podStartSLOduration=1.899169792 podStartE2EDuration="1.899169792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:40.898081182 +0000 UTC m=+706.785987538" watchObservedRunningTime="2026-10-30 23:18:40.899169792 +0000 UTC m=+706.787076144"
Oct 30 23:18:41.160296 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:41.160212    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:41.165360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:41.165336    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:41.785431 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:41.785400    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:41.785895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:41.785451    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:41.786156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:41.786127    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:18:41.786294 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:41.786267    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:18:47.589489 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:18:47.589453    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:18:48.596647 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.596617    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj"]
Oct 30 23:18:48.600055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.600040    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj"
Oct 30 23:18:48.680780 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.680750    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxhwh\" (UniqueName: \"kubernetes.io/projected/5e4686b5-2668-4680-b215-432e012c3348-kube-api-access-kxhwh\") pod \"payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj\" (UID: \"5e4686b5-2668-4680-b215-432e012c3348\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj"
Oct 30 23:18:48.680933 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.680799    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4686b5-2668-4680-b215-432e012c3348-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj\" (UID: \"5e4686b5-2668-4680-b215-432e012c3348\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj"
Oct 30 23:18:48.703538 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.703505    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-5dc3c0-dockercfg-5gx5w\""
Oct 30 23:18:48.703747 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.703505    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-5dc3c0\""
Oct 30 23:18:48.705581 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.705541    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj"]
Oct 30 23:18:48.754441 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.754407    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc"]
Oct 30 23:18:48.757683 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.757667    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc"
Oct 30 23:18:48.782071 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.782040    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t64nb\" (UniqueName: \"kubernetes.io/projected/ca4df19e-725c-4529-8028-18615a736d0b-kube-api-access-t64nb\") pod \"payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc\" (UID: \"ca4df19e-725c-4529-8028-18615a736d0b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc"
Oct 30 23:18:48.782196 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.782160    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxhwh\" (UniqueName: \"kubernetes.io/projected/5e4686b5-2668-4680-b215-432e012c3348-kube-api-access-kxhwh\") pod \"payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj\" (UID: \"5e4686b5-2668-4680-b215-432e012c3348\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj"
Oct 30 23:18:48.782242 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.782211    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4686b5-2668-4680-b215-432e012c3348-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj\" (UID: \"5e4686b5-2668-4680-b215-432e012c3348\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj"
Oct 30 23:18:48.782292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.782244    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4df19e-725c-4529-8028-18615a736d0b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc\" (UID: \"ca4df19e-725c-4529-8028-18615a736d0b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc"
Oct 30 23:18:48.782913 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.782893    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4686b5-2668-4680-b215-432e012c3348-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj\" (UID: \"5e4686b5-2668-4680-b215-432e012c3348\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj"
Oct 30 23:18:48.791111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.791086    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc"]
Oct 30 23:18:48.839546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.839519    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxhwh\" (UniqueName: \"kubernetes.io/projected/5e4686b5-2668-4680-b215-432e012c3348-kube-api-access-kxhwh\") pod \"payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj\" (UID: \"5e4686b5-2668-4680-b215-432e012c3348\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj"
Oct 30 23:18:48.883483 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.883421    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4df19e-725c-4529-8028-18615a736d0b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc\" (UID: \"ca4df19e-725c-4529-8028-18615a736d0b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc"
Oct 30 23:18:48.883620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.883469    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t64nb\" (UniqueName: \"kubernetes.io/projected/ca4df19e-725c-4529-8028-18615a736d0b-kube-api-access-t64nb\") pod \"payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc\" (UID: \"ca4df19e-725c-4529-8028-18615a736d0b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc"
Oct 30 23:18:48.884131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.884107    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4df19e-725c-4529-8028-18615a736d0b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc\" (UID: \"ca4df19e-725c-4529-8028-18615a736d0b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc"
Oct 30 23:18:48.912444 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.912422    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj"
Oct 30 23:18:48.940092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:48.940070    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t64nb\" (UniqueName: \"kubernetes.io/projected/ca4df19e-725c-4529-8028-18615a736d0b-kube-api-access-t64nb\") pod \"payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc\" (UID: \"ca4df19e-725c-4529-8028-18615a736d0b\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc"
Oct 30 23:18:49.069461 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:49.069431    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc"
Oct 30 23:18:49.284855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:49.284825    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj"]
Oct 30 23:18:49.287425 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:18:49.287399    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e4686b5_2668_4680_b215_432e012c3348.slice/crio-772667bf05d7cf5a2fe8dd7b507ea2babb1ad0ecfaa1a26fba3287e351c75703 WatchSource:0}: Error finding container 772667bf05d7cf5a2fe8dd7b507ea2babb1ad0ecfaa1a26fba3287e351c75703: Status 404 returned error can't find the container with id 772667bf05d7cf5a2fe8dd7b507ea2babb1ad0ecfaa1a26fba3287e351c75703
Oct 30 23:18:49.436241 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:49.436218    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc"]
Oct 30 23:18:49.438743 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:18:49.438719    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca4df19e_725c_4529_8028_18615a736d0b.slice/crio-46ebc4fc54296cdbf6be9c08784131bd93be9d0aa21d67022c1d4d98725abaa3 WatchSource:0}: Error finding container 46ebc4fc54296cdbf6be9c08784131bd93be9d0aa21d67022c1d4d98725abaa3: Status 404 returned error can't find the container with id 46ebc4fc54296cdbf6be9c08784131bd93be9d0aa21d67022c1d4d98725abaa3
Oct 30 23:18:49.827100 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:49.827065    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc" event={"ID":"ca4df19e-725c-4529-8028-18615a736d0b","Type":"ContainerStarted","Data":"46ebc4fc54296cdbf6be9c08784131bd93be9d0aa21d67022c1d4d98725abaa3"}
Oct 30 23:18:49.828204 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:49.828180    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj" event={"ID":"5e4686b5-2668-4680-b215-432e012c3348","Type":"ContainerStarted","Data":"772667bf05d7cf5a2fe8dd7b507ea2babb1ad0ecfaa1a26fba3287e351c75703"}
Oct 30 23:18:52.843044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:52.843007    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc" event={"ID":"ca4df19e-725c-4529-8028-18615a736d0b","Type":"ContainerStarted","Data":"1feaad350f7a51b88115a1cf473807e516ff1ea7864fcb218093b67b4b7328bd"}
Oct 30 23:18:52.843534 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:52.843106    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc"
Oct 30 23:18:52.844503 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:52.844482    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj" event={"ID":"5e4686b5-2668-4680-b215-432e012c3348","Type":"ContainerStarted","Data":"a2d3c35f37b9d2e3fb6607c626cbf188585434e08d8f5bd7d49def7d68587fe5"}
Oct 30 23:18:52.844587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:52.844569    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj"
Oct 30 23:18:52.959887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:52.959829    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc" podStartSLOduration=2.280774888 podStartE2EDuration="4.959815084s" totalImagesPullingTime="2.679040196s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:52.958453035 +0000 UTC m=+718.846359388" watchObservedRunningTime="2026-10-30 23:18:52.959815084 +0000 UTC m=+718.847721432"
Oct 30 23:18:53.076491 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.076434    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj" podStartSLOduration=2.212931592 podStartE2EDuration="5.076418595s" totalImagesPullingTime="2.863487003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:53.075549107 +0000 UTC m=+718.963455463" watchObservedRunningTime="2026-10-30 23:18:53.076418595 +0000 UTC m=+718.964324947"
Oct 30 23:18:53.506489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.506461    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm"]
Oct 30 23:18:53.509933 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.509916    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm"
Oct 30 23:18:53.563522 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.563486    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-057303-dockercfg-v54c4\""
Oct 30 23:18:53.580954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.579975    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-057303\""
Oct 30 23:18:53.617860 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.617815    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm"]
Oct 30 23:18:53.639613 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.639552    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rd74\" (UniqueName: \"kubernetes.io/projected/b6b6065b-de7d-41b2-9733-42f56d923d59-kube-api-access-4rd74\") pod \"payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm\" (UID: \"b6b6065b-de7d-41b2-9733-42f56d923d59\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm"
Oct 30 23:18:53.639794 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.639675    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b6065b-de7d-41b2-9733-42f56d923d59-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm\" (UID: \"b6b6065b-de7d-41b2-9733-42f56d923d59\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm"
Oct 30 23:18:53.668997 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.668964    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82"]
Oct 30 23:18:53.672321 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.672299    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82"
Oct 30 23:18:53.730041 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.730012    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82"]
Oct 30 23:18:53.740630 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.740577    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fca3a8d-ef11-4d95-ba93-bcafa869e2c1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82\" (UID: \"6fca3a8d-ef11-4d95-ba93-bcafa869e2c1\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82"
Oct 30 23:18:53.740834 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.740716    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6fmp\" (UniqueName: \"kubernetes.io/projected/6fca3a8d-ef11-4d95-ba93-bcafa869e2c1-kube-api-access-w6fmp\") pod \"payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82\" (UID: \"6fca3a8d-ef11-4d95-ba93-bcafa869e2c1\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82"
Oct 30 23:18:53.740834 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.740776    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rd74\" (UniqueName: \"kubernetes.io/projected/b6b6065b-de7d-41b2-9733-42f56d923d59-kube-api-access-4rd74\") pod \"payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm\" (UID: \"b6b6065b-de7d-41b2-9733-42f56d923d59\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm"
Oct 30 23:18:53.740834 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.740817    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b6065b-de7d-41b2-9733-42f56d923d59-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm\" (UID: \"b6b6065b-de7d-41b2-9733-42f56d923d59\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm"
Oct 30 23:18:53.741803 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.741774    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b6065b-de7d-41b2-9733-42f56d923d59-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm\" (UID: \"b6b6065b-de7d-41b2-9733-42f56d923d59\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm"
Oct 30 23:18:53.800412 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.800383    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rd74\" (UniqueName: \"kubernetes.io/projected/b6b6065b-de7d-41b2-9733-42f56d923d59-kube-api-access-4rd74\") pod \"payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm\" (UID: \"b6b6065b-de7d-41b2-9733-42f56d923d59\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm"
Oct 30 23:18:53.821305 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.821281    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm"
Oct 30 23:18:53.841554 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.841523    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fca3a8d-ef11-4d95-ba93-bcafa869e2c1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82\" (UID: \"6fca3a8d-ef11-4d95-ba93-bcafa869e2c1\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82"
Oct 30 23:18:53.841691 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.841588    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6fmp\" (UniqueName: \"kubernetes.io/projected/6fca3a8d-ef11-4d95-ba93-bcafa869e2c1-kube-api-access-w6fmp\") pod \"payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82\" (UID: \"6fca3a8d-ef11-4d95-ba93-bcafa869e2c1\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82"
Oct 30 23:18:53.842251 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.842231    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fca3a8d-ef11-4d95-ba93-bcafa869e2c1-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82\" (UID: \"6fca3a8d-ef11-4d95-ba93-bcafa869e2c1\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82"
Oct 30 23:18:53.900872 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.900810    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6fmp\" (UniqueName: \"kubernetes.io/projected/6fca3a8d-ef11-4d95-ba93-bcafa869e2c1-kube-api-access-w6fmp\") pod \"payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82\" (UID: \"6fca3a8d-ef11-4d95-ba93-bcafa869e2c1\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82"
Oct 30 23:18:53.984286 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:53.984253    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82"
Oct 30 23:18:54.193219 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:54.193193    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm"]
Oct 30 23:18:54.196838 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:18:54.196801    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b6065b_de7d_41b2_9733_42f56d923d59.slice/crio-20b9e27ef41f643a6e18115b2b2edcb4fd726b20efe089e736307a23bf299081 WatchSource:0}: Error finding container 20b9e27ef41f643a6e18115b2b2edcb4fd726b20efe089e736307a23bf299081: Status 404 returned error can't find the container with id 20b9e27ef41f643a6e18115b2b2edcb4fd726b20efe089e736307a23bf299081
Oct 30 23:18:54.354758 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:54.354721    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82"]
Oct 30 23:18:54.357671 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:18:54.357641    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fca3a8d_ef11_4d95_ba93_bcafa869e2c1.slice/crio-e89d859c729afd8597b72aaee7f40e7ce448ea547c2bf688a4ff75a3b0e76f86 WatchSource:0}: Error finding container e89d859c729afd8597b72aaee7f40e7ce448ea547c2bf688a4ff75a3b0e76f86: Status 404 returned error can't find the container with id e89d859c729afd8597b72aaee7f40e7ce448ea547c2bf688a4ff75a3b0e76f86
Oct 30 23:18:54.855563 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:54.855530    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm" event={"ID":"b6b6065b-de7d-41b2-9733-42f56d923d59","Type":"ContainerStarted","Data":"636769bcd33bd31630837964dd6b66d948889f64bb604f3f309b3a3d5d3e66e3"}
Oct 30 23:18:54.855795 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:54.855581    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm" event={"ID":"b6b6065b-de7d-41b2-9733-42f56d923d59","Type":"ContainerStarted","Data":"20b9e27ef41f643a6e18115b2b2edcb4fd726b20efe089e736307a23bf299081"}
Oct 30 23:18:54.855795 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:54.855683    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm"
Oct 30 23:18:54.857372 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:54.857345    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82" event={"ID":"6fca3a8d-ef11-4d95-ba93-bcafa869e2c1","Type":"ContainerStarted","Data":"155b6086942f9dd6951d898325178e8b58cffc1ad7380d5c2779304a03dd83ff"}
Oct 30 23:18:54.857526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:54.857387    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82" event={"ID":"6fca3a8d-ef11-4d95-ba93-bcafa869e2c1","Type":"ContainerStarted","Data":"e89d859c729afd8597b72aaee7f40e7ce448ea547c2bf688a4ff75a3b0e76f86"}
Oct 30 23:18:54.857526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:54.857489    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82"
Oct 30 23:18:54.974117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:54.974066    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm" podStartSLOduration=1.974051805 podStartE2EDuration="1.974051805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:54.972372808 +0000 UTC m=+720.860279160" watchObservedRunningTime="2026-10-30 23:18:54.974051805 +0000 UTC m=+720.861958157"
Oct 30 23:18:55.087582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:55.087523    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82" podStartSLOduration=2.08750909 podStartE2EDuration="2.08750909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:55.08548888 +0000 UTC m=+720.973395233" watchObservedRunningTime="2026-10-30 23:18:55.08750909 +0000 UTC m=+720.975415442"
Oct 30 23:18:56.267407 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:56.267368    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7"]
Oct 30 23:18:56.270894 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:56.270875    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7"
Oct 30 23:18:56.322925 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:56.322898    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-aadaa6-dockercfg-ktsdn\""
Oct 30 23:18:56.364520 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:56.364487    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbhnk\" (UniqueName: \"kubernetes.io/projected/2a6cb384-8d9a-4b7d-a936-b8c556d30170-kube-api-access-kbhnk\") pod \"payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7\" (UID: \"2a6cb384-8d9a-4b7d-a936-b8c556d30170\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7"
Oct 30 23:18:56.364730 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:56.364525    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6cb384-8d9a-4b7d-a936-b8c556d30170-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7\" (UID: \"2a6cb384-8d9a-4b7d-a936-b8c556d30170\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7"
Oct 30 23:18:56.372571 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:56.372548    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-aadaa6\""
Oct 30 23:18:56.374057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:56.374036    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7"]
Oct 30 23:18:56.465344 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:56.465310    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbhnk\" (UniqueName: \"kubernetes.io/projected/2a6cb384-8d9a-4b7d-a936-b8c556d30170-kube-api-access-kbhnk\") pod \"payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7\" (UID: \"2a6cb384-8d9a-4b7d-a936-b8c556d30170\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7"
Oct 30 23:18:56.465526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:56.465362    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6cb384-8d9a-4b7d-a936-b8c556d30170-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7\" (UID: \"2a6cb384-8d9a-4b7d-a936-b8c556d30170\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7"
Oct 30 23:18:56.466087 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:56.466062    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6cb384-8d9a-4b7d-a936-b8c556d30170-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7\" (UID: \"2a6cb384-8d9a-4b7d-a936-b8c556d30170\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7"
Oct 30 23:18:56.522265 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:56.522188    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbhnk\" (UniqueName: \"kubernetes.io/projected/2a6cb384-8d9a-4b7d-a936-b8c556d30170-kube-api-access-kbhnk\") pod \"payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7\" (UID: \"2a6cb384-8d9a-4b7d-a936-b8c556d30170\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7"
Oct 30 23:18:56.583379 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:56.583333    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7"
Oct 30 23:18:56.950275 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:56.950237    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7"]
Oct 30 23:18:56.953705 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:18:56.953672    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a6cb384_8d9a_4b7d_a936_b8c556d30170.slice/crio-735bad881a4e5f85dd359dd87086f22d18da1fce78357093e00b58d0f6a8cd5f WatchSource:0}: Error finding container 735bad881a4e5f85dd359dd87086f22d18da1fce78357093e00b58d0f6a8cd5f: Status 404 returned error can't find the container with id 735bad881a4e5f85dd359dd87086f22d18da1fce78357093e00b58d0f6a8cd5f
Oct 30 23:18:57.871543 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:57.871500    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7" event={"ID":"2a6cb384-8d9a-4b7d-a936-b8c556d30170","Type":"ContainerStarted","Data":"6978d4e0aafebfc7c1913d57309fe7e93fda6e19b2e996c3829b41d82bff56a8"}
Oct 30 23:18:57.871989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:57.871572    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7"
Oct 30 23:18:57.871989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:57.871590    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7" event={"ID":"2a6cb384-8d9a-4b7d-a936-b8c556d30170","Type":"ContainerStarted","Data":"735bad881a4e5f85dd359dd87086f22d18da1fce78357093e00b58d0f6a8cd5f"}
Oct 30 23:18:57.992056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:18:57.992003    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7" podStartSLOduration=1.991988875 podStartE2EDuration="1.991988875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:18:57.990787614 +0000 UTC m=+723.878694005" watchObservedRunningTime="2026-10-30 23:18:57.991988875 +0000 UTC m=+723.879895228"
Oct 30 23:19:01.412632 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:01.412576    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd"]
Oct 30 23:19:01.416784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:01.416740    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd"
Oct 30 23:19:01.473585 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:01.473554    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd"]
Oct 30 23:19:01.511323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:01.511290    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbjkg\" (UniqueName: \"kubernetes.io/projected/7b8a41ae-c270-4b4a-b758-f4dee7dd748f-kube-api-access-lbjkg\") pod \"payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd\" (UID: \"7b8a41ae-c270-4b4a-b758-f4dee7dd748f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd"
Oct 30 23:19:01.511485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:01.511353    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8a41ae-c270-4b4a-b758-f4dee7dd748f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd\" (UID: \"7b8a41ae-c270-4b4a-b758-f4dee7dd748f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd"
Oct 30 23:19:01.521440 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:01.521416    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-3c9613\""
Oct 30 23:19:01.521555 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:01.521497    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-3c9613-dockercfg-hb49l\""
Oct 30 23:19:01.612895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:01.612861    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbjkg\" (UniqueName: \"kubernetes.io/projected/7b8a41ae-c270-4b4a-b758-f4dee7dd748f-kube-api-access-lbjkg\") pod \"payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd\" (UID: \"7b8a41ae-c270-4b4a-b758-f4dee7dd748f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd"
Oct 30 23:19:01.613065 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:01.612935    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8a41ae-c270-4b4a-b758-f4dee7dd748f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd\" (UID: \"7b8a41ae-c270-4b4a-b758-f4dee7dd748f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd"
Oct 30 23:19:01.613659 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:01.613640    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8a41ae-c270-4b4a-b758-f4dee7dd748f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd\" (UID: \"7b8a41ae-c270-4b4a-b758-f4dee7dd748f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd"
Oct 30 23:19:01.670518 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:01.670443    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbjkg\" (UniqueName: \"kubernetes.io/projected/7b8a41ae-c270-4b4a-b758-f4dee7dd748f-kube-api-access-lbjkg\") pod \"payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd\" (UID: \"7b8a41ae-c270-4b4a-b758-f4dee7dd748f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd"
Oct 30 23:19:01.733013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:01.732981    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd"
Oct 30 23:19:02.114218 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:02.114189    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd"]
Oct 30 23:19:02.117748 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:19:02.117720    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8a41ae_c270_4b4a_b758_f4dee7dd748f.slice/crio-7e80752e131f522be5eef24424e61a5365685ec935a340736dd8668bfcc9c867 WatchSource:0}: Error finding container 7e80752e131f522be5eef24424e61a5365685ec935a340736dd8668bfcc9c867: Status 404 returned error can't find the container with id 7e80752e131f522be5eef24424e61a5365685ec935a340736dd8668bfcc9c867
Oct 30 23:19:02.894504 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:02.894471    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd" event={"ID":"7b8a41ae-c270-4b4a-b758-f4dee7dd748f","Type":"ContainerStarted","Data":"7b20a1eff5a6932950059bce529e7af552520a4853ad74f31ab76f5175cd8515"}
Oct 30 23:19:02.894929 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:02.894519    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd" event={"ID":"7b8a41ae-c270-4b4a-b758-f4dee7dd748f","Type":"ContainerStarted","Data":"7e80752e131f522be5eef24424e61a5365685ec935a340736dd8668bfcc9c867"}
Oct 30 23:19:02.894929 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:02.894608    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd"
Oct 30 23:19:03.010765 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:03.010714    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd" podStartSLOduration=2.010699551 podStartE2EDuration="2.010699551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:03.009037469 +0000 UTC m=+728.896943840" watchObservedRunningTime="2026-10-30 23:19:03.010699551 +0000 UTC m=+728.898605903"
Oct 30 23:19:03.852356 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:03.852331    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj"
Oct 30 23:19:03.852527 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:03.852397    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc"
Oct 30 23:19:05.866666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:05.864798    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm"
Oct 30 23:19:05.866666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:05.864906    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82"
Oct 30 23:19:06.308244 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.308211    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t"]
Oct 30 23:19:06.311522 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.311499    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t"
Oct 30 23:19:06.357719 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.357688    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t\" (UID: \"1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t"
Oct 30 23:19:06.357854 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.357796    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb8fr\" (UniqueName: \"kubernetes.io/projected/1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8-kube-api-access-sb8fr\") pod \"payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t\" (UID: \"1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t"
Oct 30 23:19:06.383626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.374625    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-2cfb80-dockercfg-nrd8m\""
Oct 30 23:19:06.383626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.377451    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t"]
Oct 30 23:19:06.383626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.378159    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-2cfb80\""
Oct 30 23:19:06.452268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.452235    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g"]
Oct 30 23:19:06.455514 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.455498    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g"
Oct 30 23:19:06.458996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.458973    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t\" (UID: \"1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t"
Oct 30 23:19:06.459110 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.459014    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb8fr\" (UniqueName: \"kubernetes.io/projected/1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8-kube-api-access-sb8fr\") pod \"payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t\" (UID: \"1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t"
Oct 30 23:19:06.459701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.459679    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t\" (UID: \"1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t"
Oct 30 23:19:06.515669 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.515632    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g"]
Oct 30 23:19:06.519446 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.519422    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb8fr\" (UniqueName: \"kubernetes.io/projected/1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8-kube-api-access-sb8fr\") pod \"payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t\" (UID: \"1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t"
Oct 30 23:19:06.560271 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.560199    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjnsb\" (UniqueName: \"kubernetes.io/projected/a44f9d90-59b0-4e9d-b95e-e987edcb1266-kube-api-access-zjnsb\") pod \"payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g\" (UID: \"a44f9d90-59b0-4e9d-b95e-e987edcb1266\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g"
Oct 30 23:19:06.560271 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.560234    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44f9d90-59b0-4e9d-b95e-e987edcb1266-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g\" (UID: \"a44f9d90-59b0-4e9d-b95e-e987edcb1266\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g"
Oct 30 23:19:06.626059 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.626027    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t"
Oct 30 23:19:06.661505 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.661447    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44f9d90-59b0-4e9d-b95e-e987edcb1266-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g\" (UID: \"a44f9d90-59b0-4e9d-b95e-e987edcb1266\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g"
Oct 30 23:19:06.661682 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.661651    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjnsb\" (UniqueName: \"kubernetes.io/projected/a44f9d90-59b0-4e9d-b95e-e987edcb1266-kube-api-access-zjnsb\") pod \"payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g\" (UID: \"a44f9d90-59b0-4e9d-b95e-e987edcb1266\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g"
Oct 30 23:19:06.662454 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.662430    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44f9d90-59b0-4e9d-b95e-e987edcb1266-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g\" (UID: \"a44f9d90-59b0-4e9d-b95e-e987edcb1266\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g"
Oct 30 23:19:06.720027 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.719966    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjnsb\" (UniqueName: \"kubernetes.io/projected/a44f9d90-59b0-4e9d-b95e-e987edcb1266-kube-api-access-zjnsb\") pod \"payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g\" (UID: \"a44f9d90-59b0-4e9d-b95e-e987edcb1266\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g"
Oct 30 23:19:06.767978 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:06.767948    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g"
Oct 30 23:19:07.000982 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:07.000956    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t"]
Oct 30 23:19:07.002854 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:19:07.002827    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d56ab6e_b5b6_4c2e_bdf1_676d50b57fd8.slice/crio-2df545704e245c958b6f3e6a53243694a3cc51df46a2e141a51b81752453c51b WatchSource:0}: Error finding container 2df545704e245c958b6f3e6a53243694a3cc51df46a2e141a51b81752453c51b: Status 404 returned error can't find the container with id 2df545704e245c958b6f3e6a53243694a3cc51df46a2e141a51b81752453c51b
Oct 30 23:19:07.145521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:07.145490    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g"]
Oct 30 23:19:07.148297 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:19:07.148269    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44f9d90_59b0_4e9d_b95e_e987edcb1266.slice/crio-612cf5731622a6a3840e7c6d4066acada28c7028bea073df1cfaf962482b45eb WatchSource:0}: Error finding container 612cf5731622a6a3840e7c6d4066acada28c7028bea073df1cfaf962482b45eb: Status 404 returned error can't find the container with id 612cf5731622a6a3840e7c6d4066acada28c7028bea073df1cfaf962482b45eb
Oct 30 23:19:07.917277 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:07.917250    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g" event={"ID":"a44f9d90-59b0-4e9d-b95e-e987edcb1266","Type":"ContainerStarted","Data":"3f31a2b4c7ec8e0371dcfca25aa8e0185cb8fb1abadd155dde92374b7ba5b548"}
Oct 30 23:19:07.917442 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:07.917302    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g"
Oct 30 23:19:07.917442 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:07.917326    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g" event={"ID":"a44f9d90-59b0-4e9d-b95e-e987edcb1266","Type":"ContainerStarted","Data":"612cf5731622a6a3840e7c6d4066acada28c7028bea073df1cfaf962482b45eb"}
Oct 30 23:19:07.918712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:07.918689    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t" event={"ID":"1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8","Type":"ContainerStarted","Data":"0fa47ece403dbc8d54cade32915e0df883b5f2375993b7e5fd1163eaefa511b4"}
Oct 30 23:19:07.918844 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:07.918740    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t"
Oct 30 23:19:07.918844 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:07.918765    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t" event={"ID":"1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8","Type":"ContainerStarted","Data":"2df545704e245c958b6f3e6a53243694a3cc51df46a2e141a51b81752453c51b"}
Oct 30 23:19:08.040183 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:08.040115    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g" podStartSLOduration=2.040097166 podStartE2EDuration="2.040097166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:08.039215523 +0000 UTC m=+733.927121875" watchObservedRunningTime="2026-10-30 23:19:08.040097166 +0000 UTC m=+733.928003520"
Oct 30 23:19:08.882684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:08.882651    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7"
Oct 30 23:19:09.000583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:09.000535    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t" podStartSLOduration=3.000521082 podStartE2EDuration="3.000521082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:08.165511567 +0000 UTC m=+734.053417928" watchObservedRunningTime="2026-10-30 23:19:09.000521082 +0000 UTC m=+734.888427464"
Oct 30 23:19:11.364623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.357778    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl"]
Oct 30 23:19:11.376629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.376115    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl"
Oct 30 23:19:11.417223 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.417191    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl"]
Oct 30 23:19:11.418747 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.418721    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba44784b-5ea3-4e23-81a4-09e94531aceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl\" (UID: \"ba44784b-5ea3-4e23-81a4-09e94531aceb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl"
Oct 30 23:19:11.418868 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.418783    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmcwc\" (UniqueName: \"kubernetes.io/projected/ba44784b-5ea3-4e23-81a4-09e94531aceb-kube-api-access-tmcwc\") pod \"payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl\" (UID: \"ba44784b-5ea3-4e23-81a4-09e94531aceb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl"
Oct 30 23:19:11.474124 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.474087    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-989a4f-dockercfg-8dptc\""
Oct 30 23:19:11.475223 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.474456    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-989a4f\""
Oct 30 23:19:11.519583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.519555    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba44784b-5ea3-4e23-81a4-09e94531aceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl\" (UID: \"ba44784b-5ea3-4e23-81a4-09e94531aceb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl"
Oct 30 23:19:11.519794 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.519634    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmcwc\" (UniqueName: \"kubernetes.io/projected/ba44784b-5ea3-4e23-81a4-09e94531aceb-kube-api-access-tmcwc\") pod \"payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl\" (UID: \"ba44784b-5ea3-4e23-81a4-09e94531aceb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl"
Oct 30 23:19:11.520304 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.520278    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba44784b-5ea3-4e23-81a4-09e94531aceb-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl\" (UID: \"ba44784b-5ea3-4e23-81a4-09e94531aceb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl"
Oct 30 23:19:11.525086 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.525066    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs"]
Oct 30 23:19:11.528607 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.528574    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs"
Oct 30 23:19:11.579625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.569199    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs"]
Oct 30 23:19:11.591919 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.591885    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmcwc\" (UniqueName: \"kubernetes.io/projected/ba44784b-5ea3-4e23-81a4-09e94531aceb-kube-api-access-tmcwc\") pod \"payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl\" (UID: \"ba44784b-5ea3-4e23-81a4-09e94531aceb\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl"
Oct 30 23:19:11.629199 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.629117    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6416daa-de15-47d8-aca5-da1b417b6750-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs\" (UID: \"b6416daa-de15-47d8-aca5-da1b417b6750\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs"
Oct 30 23:19:11.629359 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.629216    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trbs6\" (UniqueName: \"kubernetes.io/projected/b6416daa-de15-47d8-aca5-da1b417b6750-kube-api-access-trbs6\") pod \"payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs\" (UID: \"b6416daa-de15-47d8-aca5-da1b417b6750\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs"
Oct 30 23:19:11.693037 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.693002    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl"
Oct 30 23:19:11.729899 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.729865    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6416daa-de15-47d8-aca5-da1b417b6750-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs\" (UID: \"b6416daa-de15-47d8-aca5-da1b417b6750\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs"
Oct 30 23:19:11.730060 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.729932    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trbs6\" (UniqueName: \"kubernetes.io/projected/b6416daa-de15-47d8-aca5-da1b417b6750-kube-api-access-trbs6\") pod \"payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs\" (UID: \"b6416daa-de15-47d8-aca5-da1b417b6750\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs"
Oct 30 23:19:11.730878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.730854    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6416daa-de15-47d8-aca5-da1b417b6750-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs\" (UID: \"b6416daa-de15-47d8-aca5-da1b417b6750\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs"
Oct 30 23:19:11.791825 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.790231    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trbs6\" (UniqueName: \"kubernetes.io/projected/b6416daa-de15-47d8-aca5-da1b417b6750-kube-api-access-trbs6\") pod \"payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs\" (UID: \"b6416daa-de15-47d8-aca5-da1b417b6750\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs"
Oct 30 23:19:11.841267 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:11.841228    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs"
Oct 30 23:19:12.083486 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:19:12.083449    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba44784b_5ea3_4e23_81a4_09e94531aceb.slice/crio-c8e1de54cb5213ed048813b6133f995b6dbff9c84349bb60abfd02e7b6b34f1a WatchSource:0}: Error finding container c8e1de54cb5213ed048813b6133f995b6dbff9c84349bb60abfd02e7b6b34f1a: Status 404 returned error can't find the container with id c8e1de54cb5213ed048813b6133f995b6dbff9c84349bb60abfd02e7b6b34f1a
Oct 30 23:19:12.083723 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:12.083704    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl"]
Oct 30 23:19:12.238784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:12.238759    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs"]
Oct 30 23:19:12.240607 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:19:12.240565    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6416daa_de15_47d8_aca5_da1b417b6750.slice/crio-8765b870dd7539597641f3b89cb31eb1260c45a379821e06b781d2d6a3f4b20b WatchSource:0}: Error finding container 8765b870dd7539597641f3b89cb31eb1260c45a379821e06b781d2d6a3f4b20b: Status 404 returned error can't find the container with id 8765b870dd7539597641f3b89cb31eb1260c45a379821e06b781d2d6a3f4b20b
Oct 30 23:19:12.943640 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:12.943574    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs" event={"ID":"b6416daa-de15-47d8-aca5-da1b417b6750","Type":"ContainerStarted","Data":"1fefbc4127ab01f1682ca43b2837afd38f510d94b5135c29b32e843072e0851d"}
Oct 30 23:19:12.944101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:12.943658    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs" event={"ID":"b6416daa-de15-47d8-aca5-da1b417b6750","Type":"ContainerStarted","Data":"8765b870dd7539597641f3b89cb31eb1260c45a379821e06b781d2d6a3f4b20b"}
Oct 30 23:19:12.944101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:12.943687    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs"
Oct 30 23:19:12.945217 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:12.945192    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl" event={"ID":"ba44784b-5ea3-4e23-81a4-09e94531aceb","Type":"ContainerStarted","Data":"308ab5f779e59d293c8474379980f14543ac24f520d857b3e9bc7d0326892d18"}
Oct 30 23:19:12.945321 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:12.945243    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl" event={"ID":"ba44784b-5ea3-4e23-81a4-09e94531aceb","Type":"ContainerStarted","Data":"c8e1de54cb5213ed048813b6133f995b6dbff9c84349bb60abfd02e7b6b34f1a"}
Oct 30 23:19:12.945321 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:12.945269    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl"
Oct 30 23:19:13.059388 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:13.059306    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs" podStartSLOduration=2.059287748 podStartE2EDuration="2.059287748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:13.058333761 +0000 UTC m=+738.946240137" watchObservedRunningTime="2026-10-30 23:19:13.059287748 +0000 UTC m=+738.947194104"
Oct 30 23:19:13.177351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:13.176276    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl" podStartSLOduration=2.1762383 podStartE2EDuration="2.1762383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:13.174668504 +0000 UTC m=+739.062574858" watchObservedRunningTime="2026-10-30 23:19:13.1762383 +0000 UTC m=+739.064144653"
Oct 30 23:19:13.903375 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:13.903341    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd"
Oct 30 23:19:16.965719 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:16.965686    5078 generic.go:356] "Generic (PLEG): container finished" podID="3680396f-1877-4e2e-8226-3591fb4e00e2" containerID="8e406d786f20b88e30e5f0d6d9a435d1f79a12c927d867b42f4c8c3347659db4" exitCode=0
Oct 30 23:19:16.966190 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:16.965749    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf" event={"ID":"3680396f-1877-4e2e-8226-3591fb4e00e2","Type":"ContainerDied","Data":"8e406d786f20b88e30e5f0d6d9a435d1f79a12c927d867b42f4c8c3347659db4"}
Oct 30 23:19:16.966190 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:16.966112    5078 scope.go:122] "RemoveContainer" containerID="8e406d786f20b88e30e5f0d6d9a435d1f79a12c927d867b42f4c8c3347659db4"
Oct 30 23:19:16.967359 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:16.967333    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:19:16.967417 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:16.967401    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:19:16.967448 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:16.967433    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:19:17.605818 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:19:17.605783    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:19:17.985221 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:17.985128    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf" event={"ID":"3680396f-1877-4e2e-8226-3591fb4e00e2","Type":"ContainerStarted","Data":"bae8a16b8a18a1925a68f6cc84668a36e87db5cd777b2855f80a6a284b0b422f"}
Oct 30 23:19:18.212375 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:18.212341    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:19:18.212556 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:18.212529    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 23:19:18.212665 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:18.212580    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf" podUID="3680396f-1877-4e2e-8226-3591fb4e00e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 23:19:18.925464 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:18.925433    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g"
Oct 30 23:19:18.925673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:18.925651    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t"
Oct 30 23:19:19.212742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:19.212627    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 23:19:19.212742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:19.212700    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf" podUID="3680396f-1877-4e2e-8226-3591fb4e00e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 23:19:20.213177 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:20.213141    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 23:19:20.213629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:20.213193    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf" podUID="3680396f-1877-4e2e-8226-3591fb4e00e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 23:19:21.213231 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:21.213192    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 23:19:21.213751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:21.213256    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf" podUID="3680396f-1877-4e2e-8226-3591fb4e00e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 23:19:22.212913 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:22.212874    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body=
Oct 30 23:19:22.213107 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:22.212933    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf" podUID="3680396f-1877-4e2e-8226-3591fb4e00e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused"
Oct 30 23:19:23.217211 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:23.217181    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:19:23.217587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:23.217571    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:19:23.218132 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:23.218113    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:19:23.953342 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:23.953310    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs"
Oct 30 23:19:23.953508 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:23.953386    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl"
Oct 30 23:19:27.432993 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:27.432305    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": read tcp 10.133.0.2:60600->10.133.0.28:15021: read: connection reset by peer" start-of-body=
Oct 30 23:19:27.432993 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:27.432369    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j" podUID="7136a40b-0373-4850-81c8-1345ff2b8996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": read tcp 10.133.0.2:60600->10.133.0.28:15021: read: connection reset by peer"
Oct 30 23:19:28.046707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:28.046672    5078 generic.go:356] "Generic (PLEG): container finished" podID="7136a40b-0373-4850-81c8-1345ff2b8996" containerID="1aa37ceb910cc0810fffe763e4a8a06bc4a6f437b92e79d2899ffd76445d96fb" exitCode=0
Oct 30 23:19:28.046880 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:28.046744    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j" event={"ID":"7136a40b-0373-4850-81c8-1345ff2b8996","Type":"ContainerDied","Data":"1aa37ceb910cc0810fffe763e4a8a06bc4a6f437b92e79d2899ffd76445d96fb"}
Oct 30 23:19:28.047111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:28.047098    5078 scope.go:122] "RemoveContainer" containerID="1aa37ceb910cc0810fffe763e4a8a06bc4a6f437b92e79d2899ffd76445d96fb"
Oct 30 23:19:28.048361 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:28.048337    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:19:28.048410 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:28.048401    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:19:28.048448 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:28.048432    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:19:29.053144 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:29.053100    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j" event={"ID":"7136a40b-0373-4850-81c8-1345ff2b8996","Type":"ContainerStarted","Data":"e08cd65627b38fd7d88b3c421495513b09555f483fa58caf3bb3ca7758385df3"}
Oct 30 23:19:29.170088 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:29.170049    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:19:29.170440 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:29.170417    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 23:19:29.170584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:29.170562    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j" podUID="7136a40b-0373-4850-81c8-1345ff2b8996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 23:19:30.165623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:30.164155    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 23:19:30.165623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:30.164230    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j" podUID="7136a40b-0373-4850-81c8-1345ff2b8996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 23:19:31.174057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:31.174015    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 23:19:31.174692 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:31.174659    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j" podUID="7136a40b-0373-4850-81c8-1345ff2b8996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 23:19:32.166033 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:32.165994    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 23:19:32.166308 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:32.166280    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j" podUID="7136a40b-0373-4850-81c8-1345ff2b8996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 23:19:33.181737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:33.181698    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 23:19:33.182140 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:33.181772    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j" podUID="7136a40b-0373-4850-81c8-1345ff2b8996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 23:19:34.160658 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:34.160623    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Oct 30 23:19:34.160839 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:34.160686    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j" podUID="7136a40b-0373-4850-81c8-1345ff2b8996" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Oct 30 23:19:35.164619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:35.164563    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:19:35.165087 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:35.164984    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:19:35.165738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:35.165721    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:19:39.802146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.802114    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"]
Oct 30 23:19:39.805587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.805570    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:39.851980 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.851950    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"]
Oct 30 23:19:39.855323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.855308    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:39.866872 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.866855    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-2cfb80-openshift-default-dockercfg-zk28c\""
Oct 30 23:19:39.906632 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.906581    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"]
Oct 30 23:19:39.952799 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.952768    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-data\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:39.952799 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.952801    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-credential-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:39.953015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.952840    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-token\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:39.953015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.952944    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-envoy\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:39.953103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.953069    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-credential-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:39.953168 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.953115    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-workload-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:39.953168 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.953143    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-podinfo\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:39.953168 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.953159    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-envoy\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:39.953277 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.953187    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-podinfo\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:39.953277 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.953251    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rghjv\" (UniqueName: \"kubernetes.io/projected/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-kube-api-access-rghjv\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:39.953351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.953287    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvdwp\" (UniqueName: \"kubernetes.io/projected/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-kube-api-access-dvdwp\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:39.953351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.953313    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-data\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:39.953351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.953334    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istiod-ca-cert\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:39.953462 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.953393    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istiod-ca-cert\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:39.953462 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.953410    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-workload-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:39.953462 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.953429    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-workload-certs\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:39.953462 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.953450    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-token\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:39.953705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.953468    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-workload-certs\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:39.974224 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:39.974187    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"]
Oct 30 23:19:40.055109 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055032    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-data\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.055109 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055064    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-credential-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.055109 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055095    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-token\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.055394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055121    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-envoy\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.055394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055148    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-credential-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.055394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055185    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-workload-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.055394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055214    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-podinfo\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.055394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055233    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-envoy\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.055394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055262    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-podinfo\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.055394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055287    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rghjv\" (UniqueName: \"kubernetes.io/projected/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-kube-api-access-rghjv\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.055394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055326    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvdwp\" (UniqueName: \"kubernetes.io/projected/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-kube-api-access-dvdwp\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.055394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055375    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-data\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.055892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055408    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istiod-ca-cert\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.055892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055437    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istiod-ca-cert\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.055892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055463    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-workload-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.055892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055494    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-workload-certs\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.055892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055512    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-credential-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.055892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055529    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-data\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.055892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055545    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-credential-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.055892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055584    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-token\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.055892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055650    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-workload-certs\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.055892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.055856    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-workload-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.056384 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.056005    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-workload-certs\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.056384 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.056240    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istiod-ca-cert\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.056384 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.056282    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-workload-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.056384 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.056292    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istiod-ca-cert\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.056384 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.056367    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-workload-certs\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.056702 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.056469    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-data\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.058447 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.058414    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-envoy\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.058447 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.058433    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-podinfo\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.058678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.058462    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-envoy\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.058678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.058666    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-podinfo\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.112309 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.112277    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-token\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.112475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.112332    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-token\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.112790 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.112771    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rghjv\" (UniqueName: \"kubernetes.io/projected/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-kube-api-access-rghjv\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.112882 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.112847    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvdwp\" (UniqueName: \"kubernetes.io/projected/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-kube-api-access-dvdwp\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.118640 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.118622    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:40.168069 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.168026    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:40.543975 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.543930    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"]
Oct 30 23:19:40.581911 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:19:40.581881    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c6ff7a_3974_4ca3_9580_4b83b87275e2.slice/crio-c2d0666be2d56d274ed0dd95b9989d732206cc03d9094899b8005830c3467b65 WatchSource:0}: Error finding container c2d0666be2d56d274ed0dd95b9989d732206cc03d9094899b8005830c3467b65: Status 404 returned error can't find the container with id c2d0666be2d56d274ed0dd95b9989d732206cc03d9094899b8005830c3467b65
Oct 30 23:19:40.584352 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.584323    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:19:40.584454 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.584403    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:19:40.584454 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.584448    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:19:40.614092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.614064    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"]
Oct 30 23:19:40.616107 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:19:40.616064    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d12dbb_7f9a_483f_b1f9_3eeca3e2f3d3.slice/crio-fe6ade576114b1979c24de3e169d846a4772a52b307bb79b1574d7f66e303c27 WatchSource:0}: Error finding container fe6ade576114b1979c24de3e169d846a4772a52b307bb79b1574d7f66e303c27: Status 404 returned error can't find the container with id fe6ade576114b1979c24de3e169d846a4772a52b307bb79b1574d7f66e303c27
Oct 30 23:19:40.619131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.619097    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:19:40.619294 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.619193    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:19:40.619367 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:40.619350    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:19:41.132834 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:41.132798    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56" event={"ID":"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3","Type":"ContainerStarted","Data":"d575d4473691f81507b2d29f5c707036259cb7684a80f8ddd69cd80b2507aae3"}
Oct 30 23:19:41.133315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:41.132850    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56" event={"ID":"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3","Type":"ContainerStarted","Data":"fe6ade576114b1979c24de3e169d846a4772a52b307bb79b1574d7f66e303c27"}
Oct 30 23:19:41.134271 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:41.134246    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z" event={"ID":"b8c6ff7a-3974-4ca3-9580-4b83b87275e2","Type":"ContainerStarted","Data":"7766308982c9ee8dfcd04c7a61f171bc05b87efff47aea2db9702f614981a801"}
Oct 30 23:19:41.134392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:41.134292    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z" event={"ID":"b8c6ff7a-3974-4ca3-9580-4b83b87275e2","Type":"ContainerStarted","Data":"c2d0666be2d56d274ed0dd95b9989d732206cc03d9094899b8005830c3467b65"}
Oct 30 23:19:41.171074 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:41.170991    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:41.173616 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:41.173277    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 23:19:41.173616 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:41.173334    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56" podUID="43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 23:19:41.256316 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:41.256266    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56" podStartSLOduration=2.256251314 podStartE2EDuration="2.256251314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:41.255069062 +0000 UTC m=+767.142975417" watchObservedRunningTime="2026-10-30 23:19:41.256251314 +0000 UTC m=+767.144157666"
Oct 30 23:19:42.132421 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:42.131390    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:42.132421 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:42.131551    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 23:19:42.132421 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:42.131633    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z" podUID="b8c6ff7a-3974-4ca3-9580-4b83b87275e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 23:19:42.232559 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:42.186231    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 23:19:42.301900 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:42.301852    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56" podUID="43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 23:19:43.122621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:43.122057    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 23:19:43.122621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:43.122140    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z" podUID="b8c6ff7a-3974-4ca3-9580-4b83b87275e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 23:19:43.171190 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:43.171152    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 23:19:43.171487 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:43.171457    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56" podUID="43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 23:19:44.120042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:44.120002    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 23:19:44.120700 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:44.120665    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z" podUID="b8c6ff7a-3974-4ca3-9580-4b83b87275e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 23:19:44.172307 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:44.172274    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 23:19:44.172619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:44.172557    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56" podUID="43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 23:19:45.122623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:45.122013    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 23:19:45.122623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:45.122073    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z" podUID="b8c6ff7a-3974-4ca3-9580-4b83b87275e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 23:19:45.175627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:45.169293    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 23:19:45.175627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:45.169354    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56" podUID="43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 23:19:46.121625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:46.121031    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused" start-of-body=
Oct 30 23:19:46.121625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:46.121096    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z" podUID="b8c6ff7a-3974-4ca3-9580-4b83b87275e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.39:15021/healthz/ready\": dial tcp 10.133.0.39:15021: connect: connection refused"
Oct 30 23:19:46.175629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:46.171711    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused" start-of-body=
Oct 30 23:19:46.175629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:46.171774    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56" podUID="43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": dial tcp 10.133.0.40:15021: connect: connection refused"
Oct 30 23:19:47.123756 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:47.123728    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:47.124258 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:47.124232    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:47.125066 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:47.125046    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:19:47.172871 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:47.172843    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:47.173126 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:47.173110    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:47.173795 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:47.173776    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:19:47.291931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:47.291886    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z" podStartSLOduration=8.291871354 podStartE2EDuration="8.291871354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:19:41.37995571 +0000 UTC m=+767.267862064" watchObservedRunningTime="2026-10-30 23:19:47.291871354 +0000 UTC m=+773.179777707"
Oct 30 23:19:47.593693 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:19:47.593654    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:19:51.261275 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:51.261242    5078 generic.go:356] "Generic (PLEG): container finished" podID="dbde5115-111a-42ac-a735-8be11caf2de6" containerID="fa2fff8d7432e3ecb51927a8ca1dfac7ea006c47a0069719253ee440e02fd8ea" exitCode=0
Oct 30 23:19:51.261829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:51.261315    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr" event={"ID":"dbde5115-111a-42ac-a735-8be11caf2de6","Type":"ContainerDied","Data":"fa2fff8d7432e3ecb51927a8ca1dfac7ea006c47a0069719253ee440e02fd8ea"}
Oct 30 23:19:51.261829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:51.261769    5078 scope.go:122] "RemoveContainer" containerID="fa2fff8d7432e3ecb51927a8ca1dfac7ea006c47a0069719253ee440e02fd8ea"
Oct 30 23:19:51.263044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:51.263016    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:19:51.263146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:51.263096    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:19:51.263203 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:51.263143    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:19:52.286386 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:52.286352    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr" event={"ID":"dbde5115-111a-42ac-a735-8be11caf2de6","Type":"ContainerStarted","Data":"25fd3e627eb88364728578d1997ad5f7a1aad39212362784aaf16d7ee2a9bd68"}
Oct 30 23:19:52.890184 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:52.890149    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:19:52.891224 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:52.891190    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Oct 30 23:19:52.891360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:52.891239    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr" podUID="dbde5115-111a-42ac-a735-8be11caf2de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Oct 30 23:19:53.889826 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:53.889791    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Oct 30 23:19:53.890219 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:53.889855    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr" podUID="dbde5115-111a-42ac-a735-8be11caf2de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Oct 30 23:19:54.889880 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:54.889844    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Oct 30 23:19:54.890281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:54.889900    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr" podUID="dbde5115-111a-42ac-a735-8be11caf2de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Oct 30 23:19:55.889510 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:55.889476    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Oct 30 23:19:55.889716 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:55.889538    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr" podUID="dbde5115-111a-42ac-a735-8be11caf2de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Oct 30 23:19:56.785656 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:56.785614    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:19:56.892963 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:56.892849    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Oct 30 23:19:56.895397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:56.893196    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr" podUID="dbde5115-111a-42ac-a735-8be11caf2de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Oct 30 23:19:57.893998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:57.893972    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:19:57.895089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:19:57.895070    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:20:17.589837 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:20:17.589801    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:20:21.122320 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.122287    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"]
Oct 30 23:20:21.126065 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.126048    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.229083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.229051    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-home\") pod \"simulator-w2-3c9613-kserve-56f659f86b-fgwbl\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") " pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.229248 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.229113    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffd6t\" (UniqueName: \"kubernetes.io/projected/731a9563-7dbd-47a6-80b5-4a7d48433786-kube-api-access-ffd6t\") pod \"simulator-w2-3c9613-kserve-56f659f86b-fgwbl\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") " pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.229248 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.229134    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/731a9563-7dbd-47a6-80b5-4a7d48433786-tls-certs\") pod \"simulator-w2-3c9613-kserve-56f659f86b-fgwbl\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") " pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.229352 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.229330    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-model-cache\") pod \"simulator-w2-3c9613-kserve-56f659f86b-fgwbl\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") " pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.229399 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.229371    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-dshm\") pod \"simulator-w2-3c9613-kserve-56f659f86b-fgwbl\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") " pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.229399 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.229390    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-tmp-dir\") pod \"simulator-w2-3c9613-kserve-56f659f86b-fgwbl\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") " pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.229997 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.229976    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3c9613\"/\"openshift-service-ca.crt\""
Oct 30 23:20:21.230128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.230020    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-3c9613\"/\"kube-root-ca.crt\""
Oct 30 23:20:21.230128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.230052    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3c9613\"/\"default-dockercfg-w26f4\""
Oct 30 23:20:21.230220 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.230145    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3c9613\"/\"simulator-w2-3c9613-kserve-self-signed-certs\""
Oct 30 23:20:21.231778 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.231762    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"]
Oct 30 23:20:21.330481 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.330449    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffd6t\" (UniqueName: \"kubernetes.io/projected/731a9563-7dbd-47a6-80b5-4a7d48433786-kube-api-access-ffd6t\") pod \"simulator-w2-3c9613-kserve-56f659f86b-fgwbl\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") " pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.330481 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.330483    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/731a9563-7dbd-47a6-80b5-4a7d48433786-tls-certs\") pod \"simulator-w2-3c9613-kserve-56f659f86b-fgwbl\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") " pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.330810 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.330538    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-model-cache\") pod \"simulator-w2-3c9613-kserve-56f659f86b-fgwbl\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") " pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.330810 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.330562    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-dshm\") pod \"simulator-w2-3c9613-kserve-56f659f86b-fgwbl\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") " pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.330810 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.330580    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-tmp-dir\") pod \"simulator-w2-3c9613-kserve-56f659f86b-fgwbl\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") " pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.330810 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.330718    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-home\") pod \"simulator-w2-3c9613-kserve-56f659f86b-fgwbl\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") " pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.331080 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.331050    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-home\") pod \"simulator-w2-3c9613-kserve-56f659f86b-fgwbl\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") " pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.331153 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.331078    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-tmp-dir\") pod \"simulator-w2-3c9613-kserve-56f659f86b-fgwbl\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") " pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.331153 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.331109    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-model-cache\") pod \"simulator-w2-3c9613-kserve-56f659f86b-fgwbl\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") " pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.333050 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.333024    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-dshm\") pod \"simulator-w2-3c9613-kserve-56f659f86b-fgwbl\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") " pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.333322 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.333306    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/731a9563-7dbd-47a6-80b5-4a7d48433786-tls-certs\") pod \"simulator-w2-3c9613-kserve-56f659f86b-fgwbl\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") " pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.388740 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.388677    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffd6t\" (UniqueName: \"kubernetes.io/projected/731a9563-7dbd-47a6-80b5-4a7d48433786-kube-api-access-ffd6t\") pod \"simulator-w2-3c9613-kserve-56f659f86b-fgwbl\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") " pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.437149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.437123    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:21.810675 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:21.810652    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"]
Oct 30 23:20:21.813042 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:20:21.813014    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod731a9563_7dbd_47a6_80b5_4a7d48433786.slice/crio-21c8bb0cf34fee04e75b729c921c0c3c333d8b83c66be429d5daa0c0964e7503 WatchSource:0}: Error finding container 21c8bb0cf34fee04e75b729c921c0c3c333d8b83c66be429d5daa0c0964e7503: Status 404 returned error can't find the container with id 21c8bb0cf34fee04e75b729c921c0c3c333d8b83c66be429d5daa0c0964e7503
Oct 30 23:20:22.432646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:22.432590    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" event={"ID":"731a9563-7dbd-47a6-80b5-4a7d48433786","Type":"ContainerStarted","Data":"9f0d3978956c04892b30eaddcf0119d11a904cc295b34dddd3a44532e6bf7457"}
Oct 30 23:20:22.432646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:22.432652    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" event={"ID":"731a9563-7dbd-47a6-80b5-4a7d48433786","Type":"ContainerStarted","Data":"21c8bb0cf34fee04e75b729c921c0c3c333d8b83c66be429d5daa0c0964e7503"}
Oct 30 23:20:22.438048 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:22.438032    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:22.548561 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:22.548517    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podStartSLOduration=1.5485026259999999 podStartE2EDuration="1.548502626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:20:22.547014587 +0000 UTC m=+808.434920943" watchObservedRunningTime="2026-10-30 23:20:22.548502626 +0000 UTC m=+808.436408979"
Oct 30 23:20:31.437998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:31.437969    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:31.450378 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:31.450349    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:31.486580 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:31.486550    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:20:44.524438 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.524362    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"]
Oct 30 23:20:44.527844 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.527829    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.631898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.631863    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-057303\"/\"kube-root-ca.crt\""
Oct 30 23:20:44.634664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.634649    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-057303\"/\"openshift-service-ca.crt\""
Oct 30 23:20:44.634953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.634940    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-057303\"/\"default-dockercfg-csbqq\""
Oct 30 23:20:44.635115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.635097    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-057303\"/\"premium-w0-057303-kserve-self-signed-certs\""
Oct 30 23:20:44.638900 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.638872    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"]
Oct 30 23:20:44.656251 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.656221    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-model-cache\") pod \"premium-w0-057303-kserve-567764f6b-6gssc\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") " pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.656354 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.656345    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-home\") pod \"premium-w0-057303-kserve-567764f6b-6gssc\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") " pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.656393 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.656375    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-dshm\") pod \"premium-w0-057303-kserve-567764f6b-6gssc\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") " pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.656469 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.656449    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-tmp-dir\") pod \"premium-w0-057303-kserve-567764f6b-6gssc\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") " pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.656508 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.656477    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-tls-certs\") pod \"premium-w0-057303-kserve-567764f6b-6gssc\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") " pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.656508 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.656494    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2vw7\" (UniqueName: \"kubernetes.io/projected/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-kube-api-access-j2vw7\") pod \"premium-w0-057303-kserve-567764f6b-6gssc\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") " pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.757325 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.757287    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-tmp-dir\") pod \"premium-w0-057303-kserve-567764f6b-6gssc\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") " pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.757325 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.757321    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-tls-certs\") pod \"premium-w0-057303-kserve-567764f6b-6gssc\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") " pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.757575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.757340    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2vw7\" (UniqueName: \"kubernetes.io/projected/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-kube-api-access-j2vw7\") pod \"premium-w0-057303-kserve-567764f6b-6gssc\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") " pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.757575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.757388    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-model-cache\") pod \"premium-w0-057303-kserve-567764f6b-6gssc\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") " pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.757575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.757445    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-home\") pod \"premium-w0-057303-kserve-567764f6b-6gssc\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") " pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.757575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.757471    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-dshm\") pod \"premium-w0-057303-kserve-567764f6b-6gssc\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") " pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.757833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.757764    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-tmp-dir\") pod \"premium-w0-057303-kserve-567764f6b-6gssc\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") " pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.757833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.757818    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-model-cache\") pod \"premium-w0-057303-kserve-567764f6b-6gssc\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") " pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.757903 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.757884    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-home\") pod \"premium-w0-057303-kserve-567764f6b-6gssc\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") " pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.759866 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.759846    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-dshm\") pod \"premium-w0-057303-kserve-567764f6b-6gssc\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") " pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.760016 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.759998    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-tls-certs\") pod \"premium-w0-057303-kserve-567764f6b-6gssc\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") " pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.813916 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.813890    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2vw7\" (UniqueName: \"kubernetes.io/projected/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-kube-api-access-j2vw7\") pod \"premium-w0-057303-kserve-567764f6b-6gssc\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") " pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:44.838926 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:44.838900    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:45.209427 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:45.209405    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"]
Oct 30 23:20:45.211665 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:20:45.211636    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac32e8b3_eb6f_4ee7_9f59_a548422bc43a.slice/crio-1e8b000f2e71e55515d4b48c3bd5b2a63fa7d81d20ddc1d01c242148cba2ca87 WatchSource:0}: Error finding container 1e8b000f2e71e55515d4b48c3bd5b2a63fa7d81d20ddc1d01c242148cba2ca87: Status 404 returned error can't find the container with id 1e8b000f2e71e55515d4b48c3bd5b2a63fa7d81d20ddc1d01c242148cba2ca87
Oct 30 23:20:45.548882 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:45.548845    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" event={"ID":"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a","Type":"ContainerStarted","Data":"21116669f17b83f1ead244298927743d636f6ce239d0e880279bfb6ea9f31c5f"}
Oct 30 23:20:45.549303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:45.548888    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" event={"ID":"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a","Type":"ContainerStarted","Data":"1e8b000f2e71e55515d4b48c3bd5b2a63fa7d81d20ddc1d01c242148cba2ca87"}
Oct 30 23:20:45.662991 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:45.662928    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podStartSLOduration=1.662909972 podStartE2EDuration="1.662909972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:20:45.662228784 +0000 UTC m=+831.550135133" watchObservedRunningTime="2026-10-30 23:20:45.662909972 +0000 UTC m=+831.550816325"
Oct 30 23:20:45.839731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:45.839654    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:47.590413 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:20:47.590374    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:20:54.839725 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:54.839691    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:54.852665 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:54.852583    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:55.606415 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:55.606387    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:20:57.267968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.267079    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"]
Oct 30 23:20:57.271676 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.271651    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.326689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.326656    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-5dc3c0\"/\"openshift-service-ca.crt\""
Oct 30 23:20:57.326689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.326697    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-5dc3c0\"/\"kube-root-ca.crt\""
Oct 30 23:20:57.326909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.326758    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5dc3c0\"/\"simulator-w5-5dc3c0-kserve-self-signed-certs\""
Oct 30 23:20:57.326909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.326844    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5dc3c0\"/\"default-dockercfg-jnqrc\""
Oct 30 23:20:57.328551 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.328530    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"]
Oct 30 23:20:57.381317 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.381287    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-dshm\") pod \"simulator-w5-5dc3c0-kserve-5965574944-vtsp6\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") " pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.381317 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.381320    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9f19c42-fd97-4365-8494-92afa52fa4d3-tls-certs\") pod \"simulator-w5-5dc3c0-kserve-5965574944-vtsp6\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") " pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.381643 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.381346    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-model-cache\") pod \"simulator-w5-5dc3c0-kserve-5965574944-vtsp6\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") " pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.381643 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.381547    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-tmp-dir\") pod \"simulator-w5-5dc3c0-kserve-5965574944-vtsp6\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") " pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.381773 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.381666    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9rsq\" (UniqueName: \"kubernetes.io/projected/f9f19c42-fd97-4365-8494-92afa52fa4d3-kube-api-access-s9rsq\") pod \"simulator-w5-5dc3c0-kserve-5965574944-vtsp6\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") " pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.381773 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.381729    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-home\") pod \"simulator-w5-5dc3c0-kserve-5965574944-vtsp6\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") " pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.482583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.482550    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9rsq\" (UniqueName: \"kubernetes.io/projected/f9f19c42-fd97-4365-8494-92afa52fa4d3-kube-api-access-s9rsq\") pod \"simulator-w5-5dc3c0-kserve-5965574944-vtsp6\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") " pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.482815 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.482629    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-home\") pod \"simulator-w5-5dc3c0-kserve-5965574944-vtsp6\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") " pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.482815 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.482683    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-dshm\") pod \"simulator-w5-5dc3c0-kserve-5965574944-vtsp6\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") " pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.482815 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.482700    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9f19c42-fd97-4365-8494-92afa52fa4d3-tls-certs\") pod \"simulator-w5-5dc3c0-kserve-5965574944-vtsp6\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") " pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.482815 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.482724    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-model-cache\") pod \"simulator-w5-5dc3c0-kserve-5965574944-vtsp6\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") " pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.482815 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.482798    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-tmp-dir\") pod \"simulator-w5-5dc3c0-kserve-5965574944-vtsp6\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") " pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.483114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.483094    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-home\") pod \"simulator-w5-5dc3c0-kserve-5965574944-vtsp6\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") " pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.483211 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.483189    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-tmp-dir\") pod \"simulator-w5-5dc3c0-kserve-5965574944-vtsp6\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") " pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.483281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.483210    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-model-cache\") pod \"simulator-w5-5dc3c0-kserve-5965574944-vtsp6\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") " pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.485155 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.485133    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-dshm\") pod \"simulator-w5-5dc3c0-kserve-5965574944-vtsp6\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") " pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.485408 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.485390    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9f19c42-fd97-4365-8494-92afa52fa4d3-tls-certs\") pod \"simulator-w5-5dc3c0-kserve-5965574944-vtsp6\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") " pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.541104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.541074    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9rsq\" (UniqueName: \"kubernetes.io/projected/f9f19c42-fd97-4365-8494-92afa52fa4d3-kube-api-access-s9rsq\") pod \"simulator-w5-5dc3c0-kserve-5965574944-vtsp6\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") " pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.583572 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.583521    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:20:57.961231 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:57.961207    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"]
Oct 30 23:20:57.963244 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:20:57.963220    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9f19c42_fd97_4365_8494_92afa52fa4d3.slice/crio-9f0752f099b50a993a65c6e6d6b08b3dc38326942f0c50e9034b37156eaee820 WatchSource:0}: Error finding container 9f0752f099b50a993a65c6e6d6b08b3dc38326942f0c50e9034b37156eaee820: Status 404 returned error can't find the container with id 9f0752f099b50a993a65c6e6d6b08b3dc38326942f0c50e9034b37156eaee820
Oct 30 23:20:58.612115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:58.612080    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" event={"ID":"f9f19c42-fd97-4365-8494-92afa52fa4d3","Type":"ContainerStarted","Data":"2dbc8a5b2c0a6b18be31937a973466fed6332a57ca992193330c309848b0aacb"}
Oct 30 23:20:58.612115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:58.612122    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" event={"ID":"f9f19c42-fd97-4365-8494-92afa52fa4d3","Type":"ContainerStarted","Data":"9f0752f099b50a993a65c6e6d6b08b3dc38326942f0c50e9034b37156eaee820"}
Oct 30 23:20:58.729046 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:58.728995    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podStartSLOduration=1.7289803080000001 podStartE2EDuration="1.728980308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:20:58.726647456 +0000 UTC m=+844.614553808" watchObservedRunningTime="2026-10-30 23:20:58.728980308 +0000 UTC m=+844.616886660"
Oct 30 23:20:59.584616 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:20:59.584571    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:21:07.584704 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:07.584665    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:21:07.597455 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:07.597431    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:21:07.669065 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:07.669036    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:21:17.598729 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:21:17.598673    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:21:47.592649 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:21:47.592613    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:21:51.660060 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.660023    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"]
Oct 30 23:21:51.663514 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.663497    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.716372 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.716339    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-3c9613\"/\"premium-w2-3c9613-kserve-self-signed-certs\""
Oct 30 23:21:51.742000 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.741965    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-tmp-dir\") pod \"premium-w2-3c9613-kserve-7844659cf5-wknt7\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") " pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.742164 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.742007    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-tls-certs\") pod \"premium-w2-3c9613-kserve-7844659cf5-wknt7\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") " pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.742164 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.742081    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-home\") pod \"premium-w2-3c9613-kserve-7844659cf5-wknt7\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") " pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.742164 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.742116    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-model-cache\") pod \"premium-w2-3c9613-kserve-7844659cf5-wknt7\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") " pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.742264 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.742216    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmhsl\" (UniqueName: \"kubernetes.io/projected/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-kube-api-access-lmhsl\") pod \"premium-w2-3c9613-kserve-7844659cf5-wknt7\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") " pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.742264 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.742258    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-dshm\") pod \"premium-w2-3c9613-kserve-7844659cf5-wknt7\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") " pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.767290 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.767262    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"]
Oct 30 23:21:51.842783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.842753    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-home\") pod \"premium-w2-3c9613-kserve-7844659cf5-wknt7\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") " pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.842783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.842788    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-model-cache\") pod \"premium-w2-3c9613-kserve-7844659cf5-wknt7\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") " pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.843458 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.842836    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmhsl\" (UniqueName: \"kubernetes.io/projected/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-kube-api-access-lmhsl\") pod \"premium-w2-3c9613-kserve-7844659cf5-wknt7\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") " pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.843458 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.842866    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-dshm\") pod \"premium-w2-3c9613-kserve-7844659cf5-wknt7\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") " pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.843458 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.842906    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-tmp-dir\") pod \"premium-w2-3c9613-kserve-7844659cf5-wknt7\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") " pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.843458 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.843040    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-tls-certs\") pod \"premium-w2-3c9613-kserve-7844659cf5-wknt7\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") " pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.843458 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.843226    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-model-cache\") pod \"premium-w2-3c9613-kserve-7844659cf5-wknt7\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") " pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.843458 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.843252    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-tmp-dir\") pod \"premium-w2-3c9613-kserve-7844659cf5-wknt7\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") " pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.843458 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.843225    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-home\") pod \"premium-w2-3c9613-kserve-7844659cf5-wknt7\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") " pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.845388 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.845366    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-dshm\") pod \"premium-w2-3c9613-kserve-7844659cf5-wknt7\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") " pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.845679 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.845662    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-tls-certs\") pod \"premium-w2-3c9613-kserve-7844659cf5-wknt7\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") " pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.901303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.901271    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmhsl\" (UniqueName: \"kubernetes.io/projected/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-kube-api-access-lmhsl\") pod \"premium-w2-3c9613-kserve-7844659cf5-wknt7\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") " pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:51.975409 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:51.975320    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:52.366300 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:52.366276    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"]
Oct 30 23:21:52.368341 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:21:52.368316    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1d4add7_ab33_4dbd_b9aa_b238a9c3fb67.slice/crio-fab36fffe3befd0f37da622613f358c6aa8a78bdbd387508bc6be987a33f58ea WatchSource:0}: Error finding container fab36fffe3befd0f37da622613f358c6aa8a78bdbd387508bc6be987a33f58ea: Status 404 returned error can't find the container with id fab36fffe3befd0f37da622613f358c6aa8a78bdbd387508bc6be987a33f58ea
Oct 30 23:21:52.875798 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:52.875764    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" event={"ID":"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67","Type":"ContainerStarted","Data":"4e8e4b3009db4c7fb58d883885f100f1dc0d62b59cdb00d64926e7cadbdee971"}
Oct 30 23:21:52.876294 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:52.875809    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" event={"ID":"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67","Type":"ContainerStarted","Data":"fab36fffe3befd0f37da622613f358c6aa8a78bdbd387508bc6be987a33f58ea"}
Oct 30 23:21:52.976442 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:52.976414    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:21:52.991832 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:21:52.991788    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podStartSLOduration=1.991773944 podStartE2EDuration="1.991773944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:21:52.990517559 +0000 UTC m=+898.878423913" watchObservedRunningTime="2026-10-30 23:21:52.991773944 +0000 UTC m=+898.879680297"
Oct 30 23:22:01.975730 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:01.975696    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:22:01.988391 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:01.988369    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:22:02.936437 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:02.936409    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:22:13.671237 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:13.671208    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb"]
Oct 30 23:22:13.674873 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:13.674854    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb"
Oct 30 23:22:13.731172 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:13.731143    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb"]
Oct 30 23:22:13.778760 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:13.778718    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:22:13.778760 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:13.778735    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:22:13.778760 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:13.778726    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:22:13.778968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:13.778825    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:22:13.779523 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:13.779502    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z42dt\" (UniqueName: \"kubernetes.io/projected/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-kube-api-access-z42dt\") pod \"maas-api-revoke-sub-12c3a3af-sffdb\" (UID: \"8f136b2e-6c4e-4ce3-a4ac-0900ba573099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb"
Oct 30 23:22:13.779574 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:13.779561    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12c3a3af-sffdb\" (UID: \"8f136b2e-6c4e-4ce3-a4ac-0900ba573099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb"
Oct 30 23:22:13.779682 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:13.779665    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12c3a3af-sffdb\" (UID: \"8f136b2e-6c4e-4ce3-a4ac-0900ba573099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb"
Oct 30 23:22:13.880548 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:13.880522    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12c3a3af-sffdb\" (UID: \"8f136b2e-6c4e-4ce3-a4ac-0900ba573099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb"
Oct 30 23:22:13.880718 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:13.880554    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12c3a3af-sffdb\" (UID: \"8f136b2e-6c4e-4ce3-a4ac-0900ba573099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb"
Oct 30 23:22:13.880718 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:13.880688    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z42dt\" (UniqueName: \"kubernetes.io/projected/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-kube-api-access-z42dt\") pod \"maas-api-revoke-sub-12c3a3af-sffdb\" (UID: \"8f136b2e-6c4e-4ce3-a4ac-0900ba573099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb"
Oct 30 23:22:13.881160 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:13.881129    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-12c3a3af-sffdb\" (UID: \"8f136b2e-6c4e-4ce3-a4ac-0900ba573099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb"
Oct 30 23:22:13.883104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:13.883067    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-maas-api-xks-ca\") pod \"maas-api-revoke-sub-12c3a3af-sffdb\" (UID: \"8f136b2e-6c4e-4ce3-a4ac-0900ba573099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb"
Oct 30 23:22:13.937358 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:13.937296    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z42dt\" (UniqueName: \"kubernetes.io/projected/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-kube-api-access-z42dt\") pod \"maas-api-revoke-sub-12c3a3af-sffdb\" (UID: \"8f136b2e-6c4e-4ce3-a4ac-0900ba573099\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb"
Oct 30 23:22:13.987008 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:13.986982    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb"
Oct 30 23:22:14.357625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:14.357589    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb"]
Oct 30 23:22:14.359429 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:22:14.359403    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f136b2e_6c4e_4ce3_a4ac_0900ba573099.slice/crio-1f4523408c736ff3eb5ae74a8b95c412e0150fb2f743751df4926a9fa11dc84a WatchSource:0}: Error finding container 1f4523408c736ff3eb5ae74a8b95c412e0150fb2f743751df4926a9fa11dc84a: Status 404 returned error can't find the container with id 1f4523408c736ff3eb5ae74a8b95c412e0150fb2f743751df4926a9fa11dc84a
Oct 30 23:22:14.986621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:14.986560    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb" event={"ID":"8f136b2e-6c4e-4ce3-a4ac-0900ba573099","Type":"ContainerStarted","Data":"1f4523408c736ff3eb5ae74a8b95c412e0150fb2f743751df4926a9fa11dc84a"}
Oct 30 23:22:16.891841 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:16.891806    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"]
Oct 30 23:22:16.896615 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:16.896580    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:16.957927 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:16.957895    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"]
Oct 30 23:22:17.004977 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.004950    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2cfb80\"/\"default-dockercfg-9lpks\""
Oct 30 23:22:17.005299 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.005283    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2cfb80\"/\"kube-root-ca.crt\""
Oct 30 23:22:17.005346 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.005301    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-2cfb80\"/\"openshift-service-ca.crt\""
Oct 30 23:22:17.005378 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.005367    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2cfb80\"/\"premium-w3-2cfb80-kserve-self-signed-certs\""
Oct 30 23:22:17.009840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.009817    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22686b57-54a7-4206-9254-65d69fd621db-tls-certs\") pod \"premium-w3-2cfb80-kserve-b889b699d-xcpsb\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") " pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.009954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.009851    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsqgd\" (UniqueName: \"kubernetes.io/projected/22686b57-54a7-4206-9254-65d69fd621db-kube-api-access-fsqgd\") pod \"premium-w3-2cfb80-kserve-b889b699d-xcpsb\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") " pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.010018 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.009956    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-home\") pod \"premium-w3-2cfb80-kserve-b889b699d-xcpsb\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") " pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.010018 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.010002    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-model-cache\") pod \"premium-w3-2cfb80-kserve-b889b699d-xcpsb\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") " pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.010128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.010114    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-dshm\") pod \"premium-w3-2cfb80-kserve-b889b699d-xcpsb\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") " pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.010209 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.010196    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-tmp-dir\") pod \"premium-w3-2cfb80-kserve-b889b699d-xcpsb\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") " pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.111847 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.111814    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-tmp-dir\") pod \"premium-w3-2cfb80-kserve-b889b699d-xcpsb\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") " pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.112040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.111877    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22686b57-54a7-4206-9254-65d69fd621db-tls-certs\") pod \"premium-w3-2cfb80-kserve-b889b699d-xcpsb\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") " pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.112040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.111908    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsqgd\" (UniqueName: \"kubernetes.io/projected/22686b57-54a7-4206-9254-65d69fd621db-kube-api-access-fsqgd\") pod \"premium-w3-2cfb80-kserve-b889b699d-xcpsb\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") " pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.112040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.111965    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-home\") pod \"premium-w3-2cfb80-kserve-b889b699d-xcpsb\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") " pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.112040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.111991    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-model-cache\") pod \"premium-w3-2cfb80-kserve-b889b699d-xcpsb\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") " pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.112040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.112040    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-dshm\") pod \"premium-w3-2cfb80-kserve-b889b699d-xcpsb\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") " pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.112380 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.112349    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-tmp-dir\") pod \"premium-w3-2cfb80-kserve-b889b699d-xcpsb\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") " pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.112567 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.112518    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-model-cache\") pod \"premium-w3-2cfb80-kserve-b889b699d-xcpsb\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") " pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.112780 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.112530    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-home\") pod \"premium-w3-2cfb80-kserve-b889b699d-xcpsb\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") " pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.114612 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.114573    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-dshm\") pod \"premium-w3-2cfb80-kserve-b889b699d-xcpsb\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") " pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.114887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.114868    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22686b57-54a7-4206-9254-65d69fd621db-tls-certs\") pod \"premium-w3-2cfb80-kserve-b889b699d-xcpsb\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") " pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.169085 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.169006    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsqgd\" (UniqueName: \"kubernetes.io/projected/22686b57-54a7-4206-9254-65d69fd621db-kube-api-access-fsqgd\") pod \"premium-w3-2cfb80-kserve-b889b699d-xcpsb\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") " pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.209969 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.209935    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:17.590684 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:22:17.590644    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:22:17.623188 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:17.623159    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"]
Oct 30 23:22:17.626335 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:22:17.626302    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22686b57_54a7_4206_9254_65d69fd621db.slice/crio-3b4ab65e2cbc7f52daa9a7e761d79d113f0456855856bdda72a510c20d395145 WatchSource:0}: Error finding container 3b4ab65e2cbc7f52daa9a7e761d79d113f0456855856bdda72a510c20d395145: Status 404 returned error can't find the container with id 3b4ab65e2cbc7f52daa9a7e761d79d113f0456855856bdda72a510c20d395145
Oct 30 23:22:18.002903 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:18.002813    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" event={"ID":"22686b57-54a7-4206-9254-65d69fd621db","Type":"ContainerStarted","Data":"5813542ea2cb3bb73552cc199db24537fe5bae9445e9c3ad615f7bc2d80cd245"}
Oct 30 23:22:18.002903 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:18.002857    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" event={"ID":"22686b57-54a7-4206-9254-65d69fd621db","Type":"ContainerStarted","Data":"3b4ab65e2cbc7f52daa9a7e761d79d113f0456855856bdda72a510c20d395145"}
Oct 30 23:22:18.128162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:18.128097    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podStartSLOduration=2.128077873 podStartE2EDuration="2.128077873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:22:18.126911303 +0000 UTC m=+924.014817657" watchObservedRunningTime="2026-10-30 23:22:18.128077873 +0000 UTC m=+924.015984226"
Oct 30 23:22:18.210906 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:18.210867    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:20.013743 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:20.013670    5078 generic.go:356] "Generic (PLEG): container finished" podID="8f136b2e-6c4e-4ce3-a4ac-0900ba573099" containerID="41de6460f6b68ce2be6701f5016721b499c97aa90c7b72e5e32e147fd8909f0e" exitCode=0
Oct 30 23:22:20.014081 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:20.013758    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb" event={"ID":"8f136b2e-6c4e-4ce3-a4ac-0900ba573099","Type":"ContainerDied","Data":"41de6460f6b68ce2be6701f5016721b499c97aa90c7b72e5e32e147fd8909f0e"}
Oct 30 23:22:21.157178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:21.157155    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb"
Oct 30 23:22:21.287732 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:21.287660    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-kube-api-access-z42dt\" (UniqueName: \"kubernetes.io/projected/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-kube-api-access-z42dt\") pod \"8f136b2e-6c4e-4ce3-a4ac-0900ba573099\" (UID: \"8f136b2e-6c4e-4ce3-a4ac-0900ba573099\") "
Oct 30 23:22:21.287732 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:21.287727    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-maas-api-openshift-ca\") pod \"8f136b2e-6c4e-4ce3-a4ac-0900ba573099\" (UID: \"8f136b2e-6c4e-4ce3-a4ac-0900ba573099\") "
Oct 30 23:22:21.287964 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:21.287768    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-maas-api-xks-ca\") pod \"8f136b2e-6c4e-4ce3-a4ac-0900ba573099\" (UID: \"8f136b2e-6c4e-4ce3-a4ac-0900ba573099\") "
Oct 30 23:22:21.288143 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:21.288111    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-maas-api-openshift-ca" pod "8f136b2e-6c4e-4ce3-a4ac-0900ba573099" (UID: "8f136b2e-6c4e-4ce3-a4ac-0900ba573099"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:22:21.290045 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:21.290017    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-maas-api-xks-ca" pod "8f136b2e-6c4e-4ce3-a4ac-0900ba573099" (UID: "8f136b2e-6c4e-4ce3-a4ac-0900ba573099"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:22:21.290151 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:21.290046    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-kube-api-access-z42dt" pod "8f136b2e-6c4e-4ce3-a4ac-0900ba573099" (UID: "8f136b2e-6c4e-4ce3-a4ac-0900ba573099"). InnerVolumeSpecName "kube-api-access-z42dt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:22:21.389117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:21.389080    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z42dt\" (UniqueName: \"kubernetes.io/projected/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-kube-api-access-z42dt\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:22:21.389117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:21.389110    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:22:21.389117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:21.389121    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f136b2e-6c4e-4ce3-a4ac-0900ba573099-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:22:22.023639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:22.023611    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb"
Oct 30 23:22:22.023639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:22.023618    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb" event={"ID":"8f136b2e-6c4e-4ce3-a4ac-0900ba573099","Type":"ContainerDied","Data":"1f4523408c736ff3eb5ae74a8b95c412e0150fb2f743751df4926a9fa11dc84a"}
Oct 30 23:22:22.023862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:22.023671    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f4523408c736ff3eb5ae74a8b95c412e0150fb2f743751df4926a9fa11dc84a"
Oct 30 23:22:27.210283 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:27.210244    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:27.222571 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:27.222546    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:28.063262 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:28.063233    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:22:29.132169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.132138    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8"]
Oct 30 23:22:29.132544 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.132521    5078 cpu_manager.go:414] "containerMap: removing container" podUID="8f136b2e-6c4e-4ce3-a4ac-0900ba573099" containerName="revoke-keys"
Oct 30 23:22:29.132544 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.132532    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f136b2e-6c4e-4ce3-a4ac-0900ba573099" containerName="revoke-keys"
Oct 30 23:22:29.132655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.132634    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f136b2e-6c4e-4ce3-a4ac-0900ba573099" containerName="revoke-keys"
Oct 30 23:22:29.136003 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.135987    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8"
Oct 30 23:22:29.165115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.165091    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db094b09-fcd8-46dc-96f1-674cdd130756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b360db24-b5dz8\" (UID: \"db094b09-fcd8-46dc-96f1-674cdd130756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8"
Oct 30 23:22:29.165245 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.165120    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89p7k\" (UniqueName: \"kubernetes.io/projected/db094b09-fcd8-46dc-96f1-674cdd130756-kube-api-access-89p7k\") pod \"maas-api-revoke-sub-b360db24-b5dz8\" (UID: \"db094b09-fcd8-46dc-96f1-674cdd130756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8"
Oct 30 23:22:29.165245 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.165145    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db094b09-fcd8-46dc-96f1-674cdd130756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b360db24-b5dz8\" (UID: \"db094b09-fcd8-46dc-96f1-674cdd130756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8"
Oct 30 23:22:29.192738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.192712    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8"]
Oct 30 23:22:29.240026 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.239997    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:22:29.240026 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.240003    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:22:29.240026 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.240030    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:22:29.240241 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.240052    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:22:29.266319 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.266291    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db094b09-fcd8-46dc-96f1-674cdd130756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b360db24-b5dz8\" (UID: \"db094b09-fcd8-46dc-96f1-674cdd130756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8"
Oct 30 23:22:29.266485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.266324    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89p7k\" (UniqueName: \"kubernetes.io/projected/db094b09-fcd8-46dc-96f1-674cdd130756-kube-api-access-89p7k\") pod \"maas-api-revoke-sub-b360db24-b5dz8\" (UID: \"db094b09-fcd8-46dc-96f1-674cdd130756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8"
Oct 30 23:22:29.266485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.266354    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db094b09-fcd8-46dc-96f1-674cdd130756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b360db24-b5dz8\" (UID: \"db094b09-fcd8-46dc-96f1-674cdd130756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8"
Oct 30 23:22:29.267055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.267036    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db094b09-fcd8-46dc-96f1-674cdd130756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b360db24-b5dz8\" (UID: \"db094b09-fcd8-46dc-96f1-674cdd130756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8"
Oct 30 23:22:29.268817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.268800    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db094b09-fcd8-46dc-96f1-674cdd130756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b360db24-b5dz8\" (UID: \"db094b09-fcd8-46dc-96f1-674cdd130756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8"
Oct 30 23:22:29.323627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.323606    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89p7k\" (UniqueName: \"kubernetes.io/projected/db094b09-fcd8-46dc-96f1-674cdd130756-kube-api-access-89p7k\") pod \"maas-api-revoke-sub-b360db24-b5dz8\" (UID: \"db094b09-fcd8-46dc-96f1-674cdd130756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8"
Oct 30 23:22:29.448880 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.448811    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8"
Oct 30 23:22:29.818069 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:29.818046    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8"]
Oct 30 23:22:29.820351 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:22:29.820322    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb094b09_fcd8_46dc_96f1_674cdd130756.slice/crio-b03c1702643c787ce67937e4fbc0acb3274156f5cff28567aa0efa6659021127 WatchSource:0}: Error finding container b03c1702643c787ce67937e4fbc0acb3274156f5cff28567aa0efa6659021127: Status 404 returned error can't find the container with id b03c1702643c787ce67937e4fbc0acb3274156f5cff28567aa0efa6659021127
Oct 30 23:22:30.064182 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:30.064153    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8" event={"ID":"db094b09-fcd8-46dc-96f1-674cdd130756","Type":"ContainerStarted","Data":"b03c1702643c787ce67937e4fbc0acb3274156f5cff28567aa0efa6659021127"}
Oct 30 23:22:31.070379 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:31.070348    5078 generic.go:356] "Generic (PLEG): container finished" podID="db094b09-fcd8-46dc-96f1-674cdd130756" containerID="e243e07ee5d344d5b86901584b272c3fed5d72c78f06b3492bc9e3fc71033ff2" exitCode=0
Oct 30 23:22:31.070819 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:31.070435    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8" event={"ID":"db094b09-fcd8-46dc-96f1-674cdd130756","Type":"ContainerDied","Data":"e243e07ee5d344d5b86901584b272c3fed5d72c78f06b3492bc9e3fc71033ff2"}
Oct 30 23:22:32.210290 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:32.210269    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8"
Oct 30 23:22:32.295003 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:32.294975    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db094b09-fcd8-46dc-96f1-674cdd130756-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db094b09-fcd8-46dc-96f1-674cdd130756-maas-api-openshift-ca\") pod \"db094b09-fcd8-46dc-96f1-674cdd130756\" (UID: \"db094b09-fcd8-46dc-96f1-674cdd130756\") "
Oct 30 23:22:32.295187 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:32.295066    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db094b09-fcd8-46dc-96f1-674cdd130756-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db094b09-fcd8-46dc-96f1-674cdd130756-maas-api-xks-ca\") pod \"db094b09-fcd8-46dc-96f1-674cdd130756\" (UID: \"db094b09-fcd8-46dc-96f1-674cdd130756\") "
Oct 30 23:22:32.295253 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:32.295211    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db094b09-fcd8-46dc-96f1-674cdd130756-kube-api-access-89p7k\" (UniqueName: \"kubernetes.io/projected/db094b09-fcd8-46dc-96f1-674cdd130756-kube-api-access-89p7k\") pod \"db094b09-fcd8-46dc-96f1-674cdd130756\" (UID: \"db094b09-fcd8-46dc-96f1-674cdd130756\") "
Oct 30 23:22:32.295380 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:32.295352    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db094b09-fcd8-46dc-96f1-674cdd130756-maas-api-openshift-ca" pod "db094b09-fcd8-46dc-96f1-674cdd130756" (UID: "db094b09-fcd8-46dc-96f1-674cdd130756"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:22:32.295551 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:32.295533    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db094b09-fcd8-46dc-96f1-674cdd130756-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:22:32.297304 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:32.297282    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db094b09-fcd8-46dc-96f1-674cdd130756-maas-api-xks-ca" pod "db094b09-fcd8-46dc-96f1-674cdd130756" (UID: "db094b09-fcd8-46dc-96f1-674cdd130756"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:22:32.297442 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:32.297425    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db094b09-fcd8-46dc-96f1-674cdd130756-kube-api-access-89p7k" pod "db094b09-fcd8-46dc-96f1-674cdd130756" (UID: "db094b09-fcd8-46dc-96f1-674cdd130756"). InnerVolumeSpecName "kube-api-access-89p7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:22:32.396672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:32.396614    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db094b09-fcd8-46dc-96f1-674cdd130756-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:22:32.396672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:32.396640    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89p7k\" (UniqueName: \"kubernetes.io/projected/db094b09-fcd8-46dc-96f1-674cdd130756-kube-api-access-89p7k\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:22:33.081725 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:33.081689    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8" event={"ID":"db094b09-fcd8-46dc-96f1-674cdd130756","Type":"ContainerDied","Data":"b03c1702643c787ce67937e4fbc0acb3274156f5cff28567aa0efa6659021127"}
Oct 30 23:22:33.081725 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:33.081717    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8"
Oct 30 23:22:33.081935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:33.081745    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b03c1702643c787ce67937e4fbc0acb3274156f5cff28567aa0efa6659021127"
Oct 30 23:22:41.561802 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.561770    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"]
Oct 30 23:22:41.562215 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.562169    5078 cpu_manager.go:414] "containerMap: removing container" podUID="db094b09-fcd8-46dc-96f1-674cdd130756" containerName="revoke-keys"
Oct 30 23:22:41.562215 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.562179    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db094b09-fcd8-46dc-96f1-674cdd130756" containerName="revoke-keys"
Oct 30 23:22:41.562292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.562265    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="db094b09-fcd8-46dc-96f1-674cdd130756" containerName="revoke-keys"
Oct 30 23:22:41.565368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.565344    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.617884 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.617846    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-989a4f-openshift-default-dockercfg-qv7mz\""
Oct 30 23:22:41.670181 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.670155    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"]
Oct 30 23:22:41.699523 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.699499    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.699708 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.699537    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-572t9\" (UniqueName: \"kubernetes.io/projected/24046e1c-b660-45a8-bb3c-d19c8b3ff167-kube-api-access-572t9\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.699753 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.699724    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.699788 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.699763    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.699821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.699800    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.699860 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.699822    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.699921 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.699906    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.699958 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.699934    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.699998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.699956    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.800539 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.800504    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.800733 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.800546    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.800733 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.800579    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.800733 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.800627    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.800925 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.800790    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-572t9\" (UniqueName: \"kubernetes.io/projected/24046e1c-b660-45a8-bb3c-d19c8b3ff167-kube-api-access-572t9\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.800925 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.800912    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.801039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.800962    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.801039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.800970    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.801039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.800988    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.801204 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.801040    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.801204 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.801068    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.801361 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.801327    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.801490 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.801389    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.801638 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.801587    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.803515 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.803495    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.804193 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.804168    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.857924 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.857853    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.858203 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.858182    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-572t9\" (UniqueName: \"kubernetes.io/projected/24046e1c-b660-45a8-bb3c-d19c8b3ff167-kube-api-access-572t9\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:41.877360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:41.877330    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:42.308642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:42.308590    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"]
Oct 30 23:22:42.312585 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:22:42.312562    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24046e1c_b660_45a8_bb3c_d19c8b3ff167.slice/crio-b71a0d47f18defc21e5498b264586e73de9fbf1ea42ff925da2560bea8db6427 WatchSource:0}: Error finding container b71a0d47f18defc21e5498b264586e73de9fbf1ea42ff925da2560bea8db6427: Status 404 returned error can't find the container with id b71a0d47f18defc21e5498b264586e73de9fbf1ea42ff925da2560bea8db6427
Oct 30 23:22:42.314675 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:42.314647    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:22:42.314728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:42.314709    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:22:42.314766 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:42.314739    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:22:43.131015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:43.130983    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm" event={"ID":"24046e1c-b660-45a8-bb3c-d19c8b3ff167","Type":"ContainerStarted","Data":"bf9ca6c268ed31d755a6a78e241b189302e5ae903fb0950bd8e808dd3590e8ca"}
Oct 30 23:22:43.131693 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:43.131031    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm" event={"ID":"24046e1c-b660-45a8-bb3c-d19c8b3ff167","Type":"ContainerStarted","Data":"b71a0d47f18defc21e5498b264586e73de9fbf1ea42ff925da2560bea8db6427"}
Oct 30 23:22:43.303005 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:43.302947    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm" podStartSLOduration=2.302930566 podStartE2EDuration="2.302930566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:22:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:22:43.30136913 +0000 UTC m=+949.189275486" watchObservedRunningTime="2026-10-30 23:22:43.302930566 +0000 UTC m=+949.190836918"
Oct 30 23:22:43.881784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:43.881735    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:43.883146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:43.883119    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 23:22:43.883268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:43.883170    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm" podUID="24046e1c-b660-45a8-bb3c-d19c8b3ff167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 23:22:44.882074 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:44.882036    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 23:22:44.882497 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:44.882104    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm" podUID="24046e1c-b660-45a8-bb3c-d19c8b3ff167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 23:22:44.977623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:44.972183    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp"]
Oct 30 23:22:44.985305 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:44.985282    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp"
Oct 30 23:22:45.031717 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:45.031688    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp"]
Oct 30 23:22:45.034893 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:45.034869    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae3a479f-7l8fp\" (UID: \"ad06fb97-40ea-4d04-9f52-89c8f2ec85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp"
Oct 30 23:22:45.035030 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:45.034924    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npjld\" (UniqueName: \"kubernetes.io/projected/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-kube-api-access-npjld\") pod \"maas-api-revoke-sub-ae3a479f-7l8fp\" (UID: \"ad06fb97-40ea-4d04-9f52-89c8f2ec85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp"
Oct 30 23:22:45.035030 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:45.035016    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae3a479f-7l8fp\" (UID: \"ad06fb97-40ea-4d04-9f52-89c8f2ec85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp"
Oct 30 23:22:45.048346 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:45.048327    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:22:45.081618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:45.080839    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:22:45.081618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:45.081470    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:22:45.081847 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:45.081632    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:22:45.135838 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:45.135773    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae3a479f-7l8fp\" (UID: \"ad06fb97-40ea-4d04-9f52-89c8f2ec85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp"
Oct 30 23:22:45.135838 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:45.135822    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npjld\" (UniqueName: \"kubernetes.io/projected/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-kube-api-access-npjld\") pod \"maas-api-revoke-sub-ae3a479f-7l8fp\" (UID: \"ad06fb97-40ea-4d04-9f52-89c8f2ec85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp"
Oct 30 23:22:45.136005 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:45.135896    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae3a479f-7l8fp\" (UID: \"ad06fb97-40ea-4d04-9f52-89c8f2ec85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp"
Oct 30 23:22:45.136385 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:45.136362    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae3a479f-7l8fp\" (UID: \"ad06fb97-40ea-4d04-9f52-89c8f2ec85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp"
Oct 30 23:22:45.138737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:45.138713    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae3a479f-7l8fp\" (UID: \"ad06fb97-40ea-4d04-9f52-89c8f2ec85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp"
Oct 30 23:22:45.198328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:45.198296    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npjld\" (UniqueName: \"kubernetes.io/projected/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-kube-api-access-npjld\") pod \"maas-api-revoke-sub-ae3a479f-7l8fp\" (UID: \"ad06fb97-40ea-4d04-9f52-89c8f2ec85db\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp"
Oct 30 23:22:45.298210 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:45.298182    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp"
Oct 30 23:22:45.688566 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:45.688542    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp"]
Oct 30 23:22:45.690057 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:22:45.690029    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad06fb97_40ea_4d04_9f52_89c8f2ec85db.slice/crio-20c4d85b97b1e68891569494b155352bea88e705ff8894f389ee0c6962f8b728 WatchSource:0}: Error finding container 20c4d85b97b1e68891569494b155352bea88e705ff8894f389ee0c6962f8b728: Status 404 returned error can't find the container with id 20c4d85b97b1e68891569494b155352bea88e705ff8894f389ee0c6962f8b728
Oct 30 23:22:45.882034 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:45.881903    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 23:22:45.882034 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:45.881988    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm" podUID="24046e1c-b660-45a8-bb3c-d19c8b3ff167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 23:22:46.150943 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:46.150910    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp" event={"ID":"ad06fb97-40ea-4d04-9f52-89c8f2ec85db","Type":"ContainerStarted","Data":"3158863374b626e91027b43deddf58154fbe9e6a8c8a68f6cf2fb56a1b2492ec"}
Oct 30 23:22:46.151106 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:46.150963    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp" event={"ID":"ad06fb97-40ea-4d04-9f52-89c8f2ec85db","Type":"ContainerStarted","Data":"20c4d85b97b1e68891569494b155352bea88e705ff8894f389ee0c6962f8b728"}
Oct 30 23:22:46.273677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:46.271364    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp" podStartSLOduration=2.271344462 podStartE2EDuration="2.271344462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:22:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:22:46.268016275 +0000 UTC m=+952.155922631" watchObservedRunningTime="2026-10-30 23:22:46.271344462 +0000 UTC m=+952.159250818"
Oct 30 23:22:46.881814 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:46.881710    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 23:22:46.881814 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:46.881777    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm" podUID="24046e1c-b660-45a8-bb3c-d19c8b3ff167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 23:22:47.183839 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:47.183798    5078 generic.go:356] "Generic (PLEG): container finished" podID="ad06fb97-40ea-4d04-9f52-89c8f2ec85db" containerID="3158863374b626e91027b43deddf58154fbe9e6a8c8a68f6cf2fb56a1b2492ec" exitCode=0
Oct 30 23:22:47.184262 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:47.183849    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp" event={"ID":"ad06fb97-40ea-4d04-9f52-89c8f2ec85db","Type":"ContainerDied","Data":"3158863374b626e91027b43deddf58154fbe9e6a8c8a68f6cf2fb56a1b2492ec"}
Oct 30 23:22:47.610651 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:22:47.610618    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:22:47.878675 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:47.878007    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body=
Oct 30 23:22:47.878675 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:47.878104    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm" podUID="24046e1c-b660-45a8-bb3c-d19c8b3ff167" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused"
Oct 30 23:22:48.330705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:48.330678    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp"
Oct 30 23:22:48.493002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:48.492913    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-kube-api-access-npjld\" (UniqueName: \"kubernetes.io/projected/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-kube-api-access-npjld\") pod \"ad06fb97-40ea-4d04-9f52-89c8f2ec85db\" (UID: \"ad06fb97-40ea-4d04-9f52-89c8f2ec85db\") "
Oct 30 23:22:48.493172 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:48.493032    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-maas-api-openshift-ca\") pod \"ad06fb97-40ea-4d04-9f52-89c8f2ec85db\" (UID: \"ad06fb97-40ea-4d04-9f52-89c8f2ec85db\") "
Oct 30 23:22:48.493172 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:48.493114    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-maas-api-xks-ca\") pod \"ad06fb97-40ea-4d04-9f52-89c8f2ec85db\" (UID: \"ad06fb97-40ea-4d04-9f52-89c8f2ec85db\") "
Oct 30 23:22:48.493638 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:48.493612    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-maas-api-openshift-ca" pod "ad06fb97-40ea-4d04-9f52-89c8f2ec85db" (UID: "ad06fb97-40ea-4d04-9f52-89c8f2ec85db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:22:48.495751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:48.495719    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-kube-api-access-npjld" pod "ad06fb97-40ea-4d04-9f52-89c8f2ec85db" (UID: "ad06fb97-40ea-4d04-9f52-89c8f2ec85db"). InnerVolumeSpecName "kube-api-access-npjld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:22:48.495751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:48.495742    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-maas-api-xks-ca" pod "ad06fb97-40ea-4d04-9f52-89c8f2ec85db" (UID: "ad06fb97-40ea-4d04-9f52-89c8f2ec85db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:22:48.594163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:48.594131    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npjld\" (UniqueName: \"kubernetes.io/projected/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-kube-api-access-npjld\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:22:48.594163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:48.594161    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:22:48.594163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:48.594171    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad06fb97-40ea-4d04-9f52-89c8f2ec85db-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:22:49.058293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:49.058261    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:49.059486 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:49.059466    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:49.060521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:49.060501    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:22:49.223662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:49.223633    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp"
Oct 30 23:22:49.240620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:49.235583    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp" event={"ID":"ad06fb97-40ea-4d04-9f52-89c8f2ec85db","Type":"ContainerDied","Data":"20c4d85b97b1e68891569494b155352bea88e705ff8894f389ee0c6962f8b728"}
Oct 30 23:22:49.240620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:22:49.235677    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20c4d85b97b1e68891569494b155352bea88e705ff8894f389ee0c6962f8b728"
Oct 30 23:23:05.246073 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.240195    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q"]
Oct 30 23:23:05.246073 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.245415    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ad06fb97-40ea-4d04-9f52-89c8f2ec85db" containerName="revoke-keys"
Oct 30 23:23:05.246073 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.245439    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad06fb97-40ea-4d04-9f52-89c8f2ec85db" containerName="revoke-keys"
Oct 30 23:23:05.246073 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.245569    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad06fb97-40ea-4d04-9f52-89c8f2ec85db" containerName="revoke-keys"
Oct 30 23:23:05.252730 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.252699    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q"
Oct 30 23:23:05.287553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.287196    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsg99\" (UniqueName: \"kubernetes.io/projected/156a09d7-39fb-4f38-964d-bc0d7ccaa341-kube-api-access-nsg99\") pod \"maas-api-revoke-sub-3dde47e9-vnc8q\" (UID: \"156a09d7-39fb-4f38-964d-bc0d7ccaa341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q"
Oct 30 23:23:05.287553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.287267    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/156a09d7-39fb-4f38-964d-bc0d7ccaa341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dde47e9-vnc8q\" (UID: \"156a09d7-39fb-4f38-964d-bc0d7ccaa341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q"
Oct 30 23:23:05.287553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.287331    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/156a09d7-39fb-4f38-964d-bc0d7ccaa341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dde47e9-vnc8q\" (UID: \"156a09d7-39fb-4f38-964d-bc0d7ccaa341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q"
Oct 30 23:23:05.309346 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.309312    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q"]
Oct 30 23:23:05.362012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.361975    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:23:05.378621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.374815    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:23:05.378621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.374894    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:23:05.378621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.374986    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:23:05.463661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.457221    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsg99\" (UniqueName: \"kubernetes.io/projected/156a09d7-39fb-4f38-964d-bc0d7ccaa341-kube-api-access-nsg99\") pod \"maas-api-revoke-sub-3dde47e9-vnc8q\" (UID: \"156a09d7-39fb-4f38-964d-bc0d7ccaa341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q"
Oct 30 23:23:05.463661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.460962    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/156a09d7-39fb-4f38-964d-bc0d7ccaa341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dde47e9-vnc8q\" (UID: \"156a09d7-39fb-4f38-964d-bc0d7ccaa341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q"
Oct 30 23:23:05.463661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.461043    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/156a09d7-39fb-4f38-964d-bc0d7ccaa341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dde47e9-vnc8q\" (UID: \"156a09d7-39fb-4f38-964d-bc0d7ccaa341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q"
Oct 30 23:23:05.463661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.463100    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/156a09d7-39fb-4f38-964d-bc0d7ccaa341-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3dde47e9-vnc8q\" (UID: \"156a09d7-39fb-4f38-964d-bc0d7ccaa341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q"
Oct 30 23:23:05.508412 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.508320    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/156a09d7-39fb-4f38-964d-bc0d7ccaa341-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3dde47e9-vnc8q\" (UID: \"156a09d7-39fb-4f38-964d-bc0d7ccaa341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q"
Oct 30 23:23:05.533534 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.533479    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsg99\" (UniqueName: \"kubernetes.io/projected/156a09d7-39fb-4f38-964d-bc0d7ccaa341-kube-api-access-nsg99\") pod \"maas-api-revoke-sub-3dde47e9-vnc8q\" (UID: \"156a09d7-39fb-4f38-964d-bc0d7ccaa341\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q"
Oct 30 23:23:05.602778 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:05.602747    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q"
Oct 30 23:23:06.044767 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:06.044740    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q"]
Oct 30 23:23:06.046612 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:23:06.046574    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156a09d7_39fb_4f38_964d_bc0d7ccaa341.slice/crio-e0ccc8352cc753a54c4f28691a2c67a1bbdf5645842aa32bb602cdf73de71506 WatchSource:0}: Error finding container e0ccc8352cc753a54c4f28691a2c67a1bbdf5645842aa32bb602cdf73de71506: Status 404 returned error can't find the container with id e0ccc8352cc753a54c4f28691a2c67a1bbdf5645842aa32bb602cdf73de71506
Oct 30 23:23:06.335169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:06.335096    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q" event={"ID":"156a09d7-39fb-4f38-964d-bc0d7ccaa341","Type":"ContainerStarted","Data":"e0ccc8352cc753a54c4f28691a2c67a1bbdf5645842aa32bb602cdf73de71506"}
Oct 30 23:23:07.343424 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:07.343391    5078 generic.go:356] "Generic (PLEG): container finished" podID="156a09d7-39fb-4f38-964d-bc0d7ccaa341" containerID="d82830e0410ded8c1f01ce1a0992c67a9025fe26ff1ca2bd4fe92a55d0438e2d" exitCode=0
Oct 30 23:23:07.343849 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:07.343448    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q" event={"ID":"156a09d7-39fb-4f38-964d-bc0d7ccaa341","Type":"ContainerDied","Data":"d82830e0410ded8c1f01ce1a0992c67a9025fe26ff1ca2bd4fe92a55d0438e2d"}
Oct 30 23:23:08.488325 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:08.488304    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q"
Oct 30 23:23:08.588338 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:08.588313    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/156a09d7-39fb-4f38-964d-bc0d7ccaa341-kube-api-access-nsg99\" (UniqueName: \"kubernetes.io/projected/156a09d7-39fb-4f38-964d-bc0d7ccaa341-kube-api-access-nsg99\") pod \"156a09d7-39fb-4f38-964d-bc0d7ccaa341\" (UID: \"156a09d7-39fb-4f38-964d-bc0d7ccaa341\") "
Oct 30 23:23:08.588484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:08.588363    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/156a09d7-39fb-4f38-964d-bc0d7ccaa341-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/156a09d7-39fb-4f38-964d-bc0d7ccaa341-maas-api-xks-ca\") pod \"156a09d7-39fb-4f38-964d-bc0d7ccaa341\" (UID: \"156a09d7-39fb-4f38-964d-bc0d7ccaa341\") "
Oct 30 23:23:08.590739 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:08.590712    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156a09d7-39fb-4f38-964d-bc0d7ccaa341-kube-api-access-nsg99" pod "156a09d7-39fb-4f38-964d-bc0d7ccaa341" (UID: "156a09d7-39fb-4f38-964d-bc0d7ccaa341"). InnerVolumeSpecName "kube-api-access-nsg99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:23:08.590845 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:08.590709    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/156a09d7-39fb-4f38-964d-bc0d7ccaa341-maas-api-xks-ca" pod "156a09d7-39fb-4f38-964d-bc0d7ccaa341" (UID: "156a09d7-39fb-4f38-964d-bc0d7ccaa341"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:23:08.689637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:08.689524    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/156a09d7-39fb-4f38-964d-bc0d7ccaa341-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/156a09d7-39fb-4f38-964d-bc0d7ccaa341-maas-api-openshift-ca\") pod \"156a09d7-39fb-4f38-964d-bc0d7ccaa341\" (UID: \"156a09d7-39fb-4f38-964d-bc0d7ccaa341\") "
Oct 30 23:23:08.689864 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:08.689839    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/156a09d7-39fb-4f38-964d-bc0d7ccaa341-maas-api-openshift-ca" pod "156a09d7-39fb-4f38-964d-bc0d7ccaa341" (UID: "156a09d7-39fb-4f38-964d-bc0d7ccaa341"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:23:08.690024 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:08.690006    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/156a09d7-39fb-4f38-964d-bc0d7ccaa341-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:08.690085 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:08.690030    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsg99\" (UniqueName: \"kubernetes.io/projected/156a09d7-39fb-4f38-964d-bc0d7ccaa341-kube-api-access-nsg99\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:08.690085 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:08.690045    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/156a09d7-39fb-4f38-964d-bc0d7ccaa341-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:09.356535 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:09.356503    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q" event={"ID":"156a09d7-39fb-4f38-964d-bc0d7ccaa341","Type":"ContainerDied","Data":"e0ccc8352cc753a54c4f28691a2c67a1bbdf5645842aa32bb602cdf73de71506"}
Oct 30 23:23:09.356535 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:09.356535    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q"
Oct 30 23:23:09.356822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:09.356552    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0ccc8352cc753a54c4f28691a2c67a1bbdf5645842aa32bb602cdf73de71506"
Oct 30 23:23:09.368507 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:09.368484    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"]
Oct 30 23:23:09.368717 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:09.368694    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd" podUID="46d4c9f2-e2ca-4f73-8edb-e532fc479059" containerName="istio-proxy" containerID="cri-o://86e1a50c943c7216ba56615f8beca4e2f562dc34efcbbd163f14bd26b321882e" gracePeriod=30
Oct 30 23:23:14.619818 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.619797    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:23:14.657296 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.657237    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d4c9f2-e2ca-4f73-8edb-e532fc479059-kube-api-access-688mx\" (UniqueName: \"kubernetes.io/projected/46d4c9f2-e2ca-4f73-8edb-e532fc479059-kube-api-access-688mx\") pod \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") "
Oct 30 23:23:14.657296 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.657291    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-token\" (UniqueName: \"kubernetes.io/projected/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-token\") pod \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") "
Oct 30 23:23:14.657482 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.657319    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-workload-certs\") pod \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") "
Oct 30 23:23:14.657482 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.657342    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-data\") pod \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") "
Oct 30 23:23:14.657482 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.657370    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-podinfo\") pod \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") "
Oct 30 23:23:14.657482 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.657430    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-envoy\") pod \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") "
Oct 30 23:23:14.657482 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.657451    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-workload-socket\") pod \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") "
Oct 30 23:23:14.657776 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.657520    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istiod-ca-cert\") pod \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") "
Oct 30 23:23:14.657776 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.657618    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-credential-socket\") pod \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\" (UID: \"46d4c9f2-e2ca-4f73-8edb-e532fc479059\") "
Oct 30 23:23:14.657884 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.657849    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-workload-socket" pod "46d4c9f2-e2ca-4f73-8edb-e532fc479059" (UID: "46d4c9f2-e2ca-4f73-8edb-e532fc479059"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:23:14.657996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.657956    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-workload-certs" pod "46d4c9f2-e2ca-4f73-8edb-e532fc479059" (UID: "46d4c9f2-e2ca-4f73-8edb-e532fc479059"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:23:14.658100 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.658048    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:14.658157 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.658074    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-credential-socket" pod "46d4c9f2-e2ca-4f73-8edb-e532fc479059" (UID: "46d4c9f2-e2ca-4f73-8edb-e532fc479059"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:23:14.658157 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.658126    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-data" pod "46d4c9f2-e2ca-4f73-8edb-e532fc479059" (UID: "46d4c9f2-e2ca-4f73-8edb-e532fc479059"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:23:14.658423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.658252    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istiod-ca-cert" pod "46d4c9f2-e2ca-4f73-8edb-e532fc479059" (UID: "46d4c9f2-e2ca-4f73-8edb-e532fc479059"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:23:14.659958 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.659929    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-podinfo" pod "46d4c9f2-e2ca-4f73-8edb-e532fc479059" (UID: "46d4c9f2-e2ca-4f73-8edb-e532fc479059"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:23:14.660066 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.659993    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-envoy" pod "46d4c9f2-e2ca-4f73-8edb-e532fc479059" (UID: "46d4c9f2-e2ca-4f73-8edb-e532fc479059"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:23:14.660267 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.660247    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d4c9f2-e2ca-4f73-8edb-e532fc479059-kube-api-access-688mx" pod "46d4c9f2-e2ca-4f73-8edb-e532fc479059" (UID: "46d4c9f2-e2ca-4f73-8edb-e532fc479059"). InnerVolumeSpecName "kube-api-access-688mx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:23:14.660329 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.660262    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-token" pod "46d4c9f2-e2ca-4f73-8edb-e532fc479059" (UID: "46d4c9f2-e2ca-4f73-8edb-e532fc479059"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:23:14.759442 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.759420    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:14.759442 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.759441    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:14.759590 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.759452    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:14.759590 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.759461    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-688mx\" (UniqueName: \"kubernetes.io/projected/46d4c9f2-e2ca-4f73-8edb-e532fc479059-kube-api-access-688mx\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:14.759590 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.759471    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:14.759590 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.759478    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:14.759590 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.759488    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:14.759590 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:14.759496    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46d4c9f2-e2ca-4f73-8edb-e532fc479059-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:15.388083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:15.388052    5078 generic.go:356] "Generic (PLEG): container finished" podID="46d4c9f2-e2ca-4f73-8edb-e532fc479059" containerID="86e1a50c943c7216ba56615f8beca4e2f562dc34efcbbd163f14bd26b321882e" exitCode=0
Oct 30 23:23:15.388256 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:15.388098    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"
Oct 30 23:23:15.388256 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:15.388116    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd" event={"ID":"46d4c9f2-e2ca-4f73-8edb-e532fc479059","Type":"ContainerDied","Data":"86e1a50c943c7216ba56615f8beca4e2f562dc34efcbbd163f14bd26b321882e"}
Oct 30 23:23:15.388256 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:15.388156    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd" event={"ID":"46d4c9f2-e2ca-4f73-8edb-e532fc479059","Type":"ContainerDied","Data":"9234fdc1d5e8a007f8dff4b8ac11114d388bb6607eea63850edd625768bbf8cc"}
Oct 30 23:23:15.388256 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:15.388179    5078 scope.go:122] "RemoveContainer" containerID="86e1a50c943c7216ba56615f8beca4e2f562dc34efcbbd163f14bd26b321882e"
Oct 30 23:23:15.397813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:15.397791    5078 scope.go:122] "RemoveContainer" containerID="86e1a50c943c7216ba56615f8beca4e2f562dc34efcbbd163f14bd26b321882e"
Oct 30 23:23:15.398117 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:23:15.398096    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e1a50c943c7216ba56615f8beca4e2f562dc34efcbbd163f14bd26b321882e\": container with ID starting with 86e1a50c943c7216ba56615f8beca4e2f562dc34efcbbd163f14bd26b321882e not found: ID does not exist" containerID="86e1a50c943c7216ba56615f8beca4e2f562dc34efcbbd163f14bd26b321882e"
Oct 30 23:23:15.398212 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:15.398124    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e1a50c943c7216ba56615f8beca4e2f562dc34efcbbd163f14bd26b321882e"} err="failed to get container status \"86e1a50c943c7216ba56615f8beca4e2f562dc34efcbbd163f14bd26b321882e\": rpc error: code = NotFound desc = could not find container \"86e1a50c943c7216ba56615f8beca4e2f562dc34efcbbd163f14bd26b321882e\": container with ID starting with 86e1a50c943c7216ba56615f8beca4e2f562dc34efcbbd163f14bd26b321882e not found: ID does not exist"
Oct 30 23:23:15.602183 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:15.602152    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"]
Oct 30 23:23:15.605189 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:15.605169    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q4jfd"]
Oct 30 23:23:16.037567 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.037536    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22"]
Oct 30 23:23:16.037989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.037975    5078 cpu_manager.go:414] "containerMap: removing container" podUID="156a09d7-39fb-4f38-964d-bc0d7ccaa341" containerName="revoke-keys"
Oct 30 23:23:16.038044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.037990    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="156a09d7-39fb-4f38-964d-bc0d7ccaa341" containerName="revoke-keys"
Oct 30 23:23:16.038044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.038018    5078 cpu_manager.go:414] "containerMap: removing container" podUID="46d4c9f2-e2ca-4f73-8edb-e532fc479059" containerName="istio-proxy"
Oct 30 23:23:16.038044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.038025    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46d4c9f2-e2ca-4f73-8edb-e532fc479059" containerName="istio-proxy"
Oct 30 23:23:16.038137 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.038098    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="46d4c9f2-e2ca-4f73-8edb-e532fc479059" containerName="istio-proxy"
Oct 30 23:23:16.038137 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.038108    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="156a09d7-39fb-4f38-964d-bc0d7ccaa341" containerName="revoke-keys"
Oct 30 23:23:16.042484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.042470    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22"
Oct 30 23:23:16.096468 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.096432    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:23:16.096625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.096534    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:23:16.096723 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.096709    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:23:16.096807 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.096793    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:23:16.097698 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.097677    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22"]
Oct 30 23:23:16.174804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.174779    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa22026-5e03-4295-8dd6-44b21b694159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c970766b-dzj22\" (UID: \"afa22026-5e03-4295-8dd6-44b21b694159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22"
Oct 30 23:23:16.174934 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.174850    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86rzt\" (UniqueName: \"kubernetes.io/projected/afa22026-5e03-4295-8dd6-44b21b694159-kube-api-access-86rzt\") pod \"maas-api-revoke-sub-c970766b-dzj22\" (UID: \"afa22026-5e03-4295-8dd6-44b21b694159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22"
Oct 30 23:23:16.175003 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.174982    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa22026-5e03-4295-8dd6-44b21b694159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c970766b-dzj22\" (UID: \"afa22026-5e03-4295-8dd6-44b21b694159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22"
Oct 30 23:23:16.276049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.276010    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa22026-5e03-4295-8dd6-44b21b694159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c970766b-dzj22\" (UID: \"afa22026-5e03-4295-8dd6-44b21b694159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22"
Oct 30 23:23:16.276256 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.276076    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86rzt\" (UniqueName: \"kubernetes.io/projected/afa22026-5e03-4295-8dd6-44b21b694159-kube-api-access-86rzt\") pod \"maas-api-revoke-sub-c970766b-dzj22\" (UID: \"afa22026-5e03-4295-8dd6-44b21b694159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22"
Oct 30 23:23:16.276256 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.276173    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa22026-5e03-4295-8dd6-44b21b694159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c970766b-dzj22\" (UID: \"afa22026-5e03-4295-8dd6-44b21b694159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22"
Oct 30 23:23:16.276744 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.276721    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa22026-5e03-4295-8dd6-44b21b694159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c970766b-dzj22\" (UID: \"afa22026-5e03-4295-8dd6-44b21b694159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22"
Oct 30 23:23:16.278712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.278695    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa22026-5e03-4295-8dd6-44b21b694159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c970766b-dzj22\" (UID: \"afa22026-5e03-4295-8dd6-44b21b694159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22"
Oct 30 23:23:16.296950 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.296921    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46d4c9f2-e2ca-4f73-8edb-e532fc479059" path="/var/lib/kubelet/pods/46d4c9f2-e2ca-4f73-8edb-e532fc479059/volumes"
Oct 30 23:23:16.333047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.333025    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86rzt\" (UniqueName: \"kubernetes.io/projected/afa22026-5e03-4295-8dd6-44b21b694159-kube-api-access-86rzt\") pod \"maas-api-revoke-sub-c970766b-dzj22\" (UID: \"afa22026-5e03-4295-8dd6-44b21b694159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22"
Oct 30 23:23:16.354701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.354679    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22"
Oct 30 23:23:16.729634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:16.726646    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22"]
Oct 30 23:23:16.731098 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:23:16.731068    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa22026_5e03_4295_8dd6_44b21b694159.slice/crio-bce3c0671f1151694f82fb63ac09b731f57b46b3325374d6e36b5114173c10b4 WatchSource:0}: Error finding container bce3c0671f1151694f82fb63ac09b731f57b46b3325374d6e36b5114173c10b4: Status 404 returned error can't find the container with id bce3c0671f1151694f82fb63ac09b731f57b46b3325374d6e36b5114173c10b4
Oct 30 23:23:17.402159 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:17.402068    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22" event={"ID":"afa22026-5e03-4295-8dd6-44b21b694159","Type":"ContainerStarted","Data":"f6f838a09165e60c2ab23d03801409cbd2cc5fa59e87c1b242c4bbf225f06a12"}
Oct 30 23:23:17.402159 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:17.402118    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22" event={"ID":"afa22026-5e03-4295-8dd6-44b21b694159","Type":"ContainerStarted","Data":"bce3c0671f1151694f82fb63ac09b731f57b46b3325374d6e36b5114173c10b4"}
Oct 30 23:23:17.519260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:17.519212    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22" podStartSLOduration=2.519199267 podStartE2EDuration="2.519199267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:23:17.516995591 +0000 UTC m=+983.404901945" watchObservedRunningTime="2026-10-30 23:23:17.519199267 +0000 UTC m=+983.407105674"
Oct 30 23:23:17.591668 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:23:17.591624    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:23:18.410028 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:18.409998    5078 generic.go:356] "Generic (PLEG): container finished" podID="afa22026-5e03-4295-8dd6-44b21b694159" containerID="f6f838a09165e60c2ab23d03801409cbd2cc5fa59e87c1b242c4bbf225f06a12" exitCode=0
Oct 30 23:23:18.410419 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:18.410070    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22" event={"ID":"afa22026-5e03-4295-8dd6-44b21b694159","Type":"ContainerDied","Data":"f6f838a09165e60c2ab23d03801409cbd2cc5fa59e87c1b242c4bbf225f06a12"}
Oct 30 23:23:19.556766 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:19.556743    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22"
Oct 30 23:23:19.608898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:19.608862    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afa22026-5e03-4295-8dd6-44b21b694159-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa22026-5e03-4295-8dd6-44b21b694159-maas-api-xks-ca\") pod \"afa22026-5e03-4295-8dd6-44b21b694159\" (UID: \"afa22026-5e03-4295-8dd6-44b21b694159\") "
Oct 30 23:23:19.609075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:19.608909    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afa22026-5e03-4295-8dd6-44b21b694159-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa22026-5e03-4295-8dd6-44b21b694159-maas-api-openshift-ca\") pod \"afa22026-5e03-4295-8dd6-44b21b694159\" (UID: \"afa22026-5e03-4295-8dd6-44b21b694159\") "
Oct 30 23:23:19.609075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:19.609052    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afa22026-5e03-4295-8dd6-44b21b694159-kube-api-access-86rzt\" (UniqueName: \"kubernetes.io/projected/afa22026-5e03-4295-8dd6-44b21b694159-kube-api-access-86rzt\") pod \"afa22026-5e03-4295-8dd6-44b21b694159\" (UID: \"afa22026-5e03-4295-8dd6-44b21b694159\") "
Oct 30 23:23:19.609351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:19.609320    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afa22026-5e03-4295-8dd6-44b21b694159-maas-api-openshift-ca" pod "afa22026-5e03-4295-8dd6-44b21b694159" (UID: "afa22026-5e03-4295-8dd6-44b21b694159"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:23:19.611363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:19.611333    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa22026-5e03-4295-8dd6-44b21b694159-kube-api-access-86rzt" pod "afa22026-5e03-4295-8dd6-44b21b694159" (UID: "afa22026-5e03-4295-8dd6-44b21b694159"). InnerVolumeSpecName "kube-api-access-86rzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:23:19.611451 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:19.611334    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afa22026-5e03-4295-8dd6-44b21b694159-maas-api-xks-ca" pod "afa22026-5e03-4295-8dd6-44b21b694159" (UID: "afa22026-5e03-4295-8dd6-44b21b694159"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:23:19.710096 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:19.710070    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86rzt\" (UniqueName: \"kubernetes.io/projected/afa22026-5e03-4295-8dd6-44b21b694159-kube-api-access-86rzt\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:19.710096 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:19.710096    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afa22026-5e03-4295-8dd6-44b21b694159-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:19.710275 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:19.710105    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afa22026-5e03-4295-8dd6-44b21b694159-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:19.925373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:19.925335    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt"]
Oct 30 23:23:19.926009 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:19.925987    5078 cpu_manager.go:414] "containerMap: removing container" podUID="afa22026-5e03-4295-8dd6-44b21b694159" containerName="revoke-keys"
Oct 30 23:23:19.926088 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:19.926011    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa22026-5e03-4295-8dd6-44b21b694159" containerName="revoke-keys"
Oct 30 23:23:19.926138 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:19.926125    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa22026-5e03-4295-8dd6-44b21b694159" containerName="revoke-keys"
Oct 30 23:23:19.930725 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:19.930669    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt"
Oct 30 23:23:19.986544 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:19.986515    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt"]
Oct 30 23:23:20.012375 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:20.012341    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3f37608-g8pqt\" (UID: \"d5c655ce-f2ca-411a-b95e-48adcd4fe77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt"
Oct 30 23:23:20.012497 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:20.012382    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjk6c\" (UniqueName: \"kubernetes.io/projected/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-kube-api-access-bjk6c\") pod \"maas-api-revoke-sub-e3f37608-g8pqt\" (UID: \"d5c655ce-f2ca-411a-b95e-48adcd4fe77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt"
Oct 30 23:23:20.012497 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:20.012414    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3f37608-g8pqt\" (UID: \"d5c655ce-f2ca-411a-b95e-48adcd4fe77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt"
Oct 30 23:23:20.113226 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:20.113190    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3f37608-g8pqt\" (UID: \"d5c655ce-f2ca-411a-b95e-48adcd4fe77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt"
Oct 30 23:23:20.113349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:20.113241    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjk6c\" (UniqueName: \"kubernetes.io/projected/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-kube-api-access-bjk6c\") pod \"maas-api-revoke-sub-e3f37608-g8pqt\" (UID: \"d5c655ce-f2ca-411a-b95e-48adcd4fe77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt"
Oct 30 23:23:20.113349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:20.113273    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3f37608-g8pqt\" (UID: \"d5c655ce-f2ca-411a-b95e-48adcd4fe77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt"
Oct 30 23:23:20.113877 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:20.113844    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3f37608-g8pqt\" (UID: \"d5c655ce-f2ca-411a-b95e-48adcd4fe77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt"
Oct 30 23:23:20.115707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:20.115690    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3f37608-g8pqt\" (UID: \"d5c655ce-f2ca-411a-b95e-48adcd4fe77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt"
Oct 30 23:23:20.170347 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:20.170322    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjk6c\" (UniqueName: \"kubernetes.io/projected/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-kube-api-access-bjk6c\") pod \"maas-api-revoke-sub-e3f37608-g8pqt\" (UID: \"d5c655ce-f2ca-411a-b95e-48adcd4fe77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt"
Oct 30 23:23:20.245366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:20.245304    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt"
Oct 30 23:23:20.421125 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:20.421089    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22" event={"ID":"afa22026-5e03-4295-8dd6-44b21b694159","Type":"ContainerDied","Data":"bce3c0671f1151694f82fb63ac09b731f57b46b3325374d6e36b5114173c10b4"}
Oct 30 23:23:20.421302 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:20.421153    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bce3c0671f1151694f82fb63ac09b731f57b46b3325374d6e36b5114173c10b4"
Oct 30 23:23:20.421302 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:20.421102    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22"
Oct 30 23:23:20.665780 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:20.665755    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt"]
Oct 30 23:23:20.667115 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:23:20.667089    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c655ce_f2ca_411a_b95e_48adcd4fe77e.slice/crio-17448facde76b89d4a4bf0dc7a88257038594cbe48b2b1df8fd813a500f28c58 WatchSource:0}: Error finding container 17448facde76b89d4a4bf0dc7a88257038594cbe48b2b1df8fd813a500f28c58: Status 404 returned error can't find the container with id 17448facde76b89d4a4bf0dc7a88257038594cbe48b2b1df8fd813a500f28c58
Oct 30 23:23:21.435177 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:21.435147    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt" event={"ID":"d5c655ce-f2ca-411a-b95e-48adcd4fe77e","Type":"ContainerStarted","Data":"0a9c92f1997d85a6e13496d478cd82857323d0d28b93d6bad15cdebd9a68807f"}
Oct 30 23:23:21.435358 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:21.435216    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt" event={"ID":"d5c655ce-f2ca-411a-b95e-48adcd4fe77e","Type":"ContainerStarted","Data":"17448facde76b89d4a4bf0dc7a88257038594cbe48b2b1df8fd813a500f28c58"}
Oct 30 23:23:21.550808 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:21.550755    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt" podStartSLOduration=2.550740613 podStartE2EDuration="2.550740613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:23:21.548384749 +0000 UTC m=+987.436291103" watchObservedRunningTime="2026-10-30 23:23:21.550740613 +0000 UTC m=+987.438646966"
Oct 30 23:23:22.441668 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:22.441629    5078 generic.go:356] "Generic (PLEG): container finished" podID="d5c655ce-f2ca-411a-b95e-48adcd4fe77e" containerID="0a9c92f1997d85a6e13496d478cd82857323d0d28b93d6bad15cdebd9a68807f" exitCode=0
Oct 30 23:23:22.442128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:22.441764    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt" event={"ID":"d5c655ce-f2ca-411a-b95e-48adcd4fe77e","Type":"ContainerDied","Data":"0a9c92f1997d85a6e13496d478cd82857323d0d28b93d6bad15cdebd9a68807f"}
Oct 30 23:23:23.573807 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:23.573784    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt"
Oct 30 23:23:23.651959 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:23.651933    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-maas-api-xks-ca\") pod \"d5c655ce-f2ca-411a-b95e-48adcd4fe77e\" (UID: \"d5c655ce-f2ca-411a-b95e-48adcd4fe77e\") "
Oct 30 23:23:23.652086 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:23.651982    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-maas-api-openshift-ca\") pod \"d5c655ce-f2ca-411a-b95e-48adcd4fe77e\" (UID: \"d5c655ce-f2ca-411a-b95e-48adcd4fe77e\") "
Oct 30 23:23:23.652086 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:23.652019    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-kube-api-access-bjk6c\" (UniqueName: \"kubernetes.io/projected/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-kube-api-access-bjk6c\") pod \"d5c655ce-f2ca-411a-b95e-48adcd4fe77e\" (UID: \"d5c655ce-f2ca-411a-b95e-48adcd4fe77e\") "
Oct 30 23:23:23.652313 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:23.652291    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-maas-api-openshift-ca" pod "d5c655ce-f2ca-411a-b95e-48adcd4fe77e" (UID: "d5c655ce-f2ca-411a-b95e-48adcd4fe77e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:23:23.654119 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:23.654098    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-maas-api-xks-ca" pod "d5c655ce-f2ca-411a-b95e-48adcd4fe77e" (UID: "d5c655ce-f2ca-411a-b95e-48adcd4fe77e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:23:23.654245 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:23.654223    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-kube-api-access-bjk6c" pod "d5c655ce-f2ca-411a-b95e-48adcd4fe77e" (UID: "d5c655ce-f2ca-411a-b95e-48adcd4fe77e"). InnerVolumeSpecName "kube-api-access-bjk6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:23:23.753249 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:23.753170    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:23.753249 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:23.753201    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjk6c\" (UniqueName: \"kubernetes.io/projected/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-kube-api-access-bjk6c\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:23.753249 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:23.753215    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5c655ce-f2ca-411a-b95e-48adcd4fe77e-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:24.453127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.453103    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt"
Oct 30 23:23:24.453127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.453115    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt" event={"ID":"d5c655ce-f2ca-411a-b95e-48adcd4fe77e","Type":"ContainerDied","Data":"17448facde76b89d4a4bf0dc7a88257038594cbe48b2b1df8fd813a500f28c58"}
Oct 30 23:23:24.453334 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.453162    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17448facde76b89d4a4bf0dc7a88257038594cbe48b2b1df8fd813a500f28c58"
Oct 30 23:23:24.757823 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.757749    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m"]
Oct 30 23:23:24.758204 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.758189    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d5c655ce-f2ca-411a-b95e-48adcd4fe77e" containerName="revoke-keys"
Oct 30 23:23:24.758260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.758205    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c655ce-f2ca-411a-b95e-48adcd4fe77e" containerName="revoke-keys"
Oct 30 23:23:24.758298 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.758288    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c655ce-f2ca-411a-b95e-48adcd4fe77e" containerName="revoke-keys"
Oct 30 23:23:24.761357 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.761343    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m"
Oct 30 23:23:24.817470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.817425    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:23:24.817965 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.817943    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:23:24.820813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.820791    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m"]
Oct 30 23:23:24.865940 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.865911    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5vns\" (UniqueName: \"kubernetes.io/projected/98a11942-14ba-4757-8264-01bf9d03db99-kube-api-access-k5vns\") pod \"maas-api-revoke-sub-0a45668e-prx6m\" (UID: \"98a11942-14ba-4757-8264-01bf9d03db99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m"
Oct 30 23:23:24.866066 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.865982    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a11942-14ba-4757-8264-01bf9d03db99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a45668e-prx6m\" (UID: \"98a11942-14ba-4757-8264-01bf9d03db99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m"
Oct 30 23:23:24.866066 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.866055    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a11942-14ba-4757-8264-01bf9d03db99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a45668e-prx6m\" (UID: \"98a11942-14ba-4757-8264-01bf9d03db99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m"
Oct 30 23:23:24.866557 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.866539    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:23:24.866591 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.866573    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:23:24.966789 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.966762    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5vns\" (UniqueName: \"kubernetes.io/projected/98a11942-14ba-4757-8264-01bf9d03db99-kube-api-access-k5vns\") pod \"maas-api-revoke-sub-0a45668e-prx6m\" (UID: \"98a11942-14ba-4757-8264-01bf9d03db99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m"
Oct 30 23:23:24.966949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.966830    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a11942-14ba-4757-8264-01bf9d03db99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a45668e-prx6m\" (UID: \"98a11942-14ba-4757-8264-01bf9d03db99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m"
Oct 30 23:23:24.966949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.966863    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a11942-14ba-4757-8264-01bf9d03db99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a45668e-prx6m\" (UID: \"98a11942-14ba-4757-8264-01bf9d03db99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m"
Oct 30 23:23:24.967525 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.967499    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a11942-14ba-4757-8264-01bf9d03db99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a45668e-prx6m\" (UID: \"98a11942-14ba-4757-8264-01bf9d03db99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m"
Oct 30 23:23:24.969360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:24.969332    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a11942-14ba-4757-8264-01bf9d03db99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a45668e-prx6m\" (UID: \"98a11942-14ba-4757-8264-01bf9d03db99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m"
Oct 30 23:23:25.025761 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:25.025700    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5vns\" (UniqueName: \"kubernetes.io/projected/98a11942-14ba-4757-8264-01bf9d03db99-kube-api-access-k5vns\") pod \"maas-api-revoke-sub-0a45668e-prx6m\" (UID: \"98a11942-14ba-4757-8264-01bf9d03db99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m"
Oct 30 23:23:25.073859 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:25.073834    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m"
Oct 30 23:23:25.442743 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:25.442716    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m"]
Oct 30 23:23:25.444802 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:23:25.444776    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a11942_14ba_4757_8264_01bf9d03db99.slice/crio-1d30a60e62e436471bdf3843132bb50b598f012a052c972676f55aeecd90aebe WatchSource:0}: Error finding container 1d30a60e62e436471bdf3843132bb50b598f012a052c972676f55aeecd90aebe: Status 404 returned error can't find the container with id 1d30a60e62e436471bdf3843132bb50b598f012a052c972676f55aeecd90aebe
Oct 30 23:23:25.459453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:25.459425    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m" event={"ID":"98a11942-14ba-4757-8264-01bf9d03db99","Type":"ContainerStarted","Data":"1d30a60e62e436471bdf3843132bb50b598f012a052c972676f55aeecd90aebe"}
Oct 30 23:23:26.467078 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:26.467044    5078 generic.go:356] "Generic (PLEG): container finished" podID="98a11942-14ba-4757-8264-01bf9d03db99" containerID="6dd2230b2c104860eb05092ffa07d36dd1ea033f2b50dcb95fa3cf86ff546737" exitCode=0
Oct 30 23:23:26.467437 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:26.467083    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m" event={"ID":"98a11942-14ba-4757-8264-01bf9d03db99","Type":"ContainerDied","Data":"6dd2230b2c104860eb05092ffa07d36dd1ea033f2b50dcb95fa3cf86ff546737"}
Oct 30 23:23:27.611728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:27.611704    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m"
Oct 30 23:23:27.698626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:27.698581    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98a11942-14ba-4757-8264-01bf9d03db99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a11942-14ba-4757-8264-01bf9d03db99-maas-api-openshift-ca\") pod \"98a11942-14ba-4757-8264-01bf9d03db99\" (UID: \"98a11942-14ba-4757-8264-01bf9d03db99\") "
Oct 30 23:23:27.698788 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:27.698751    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98a11942-14ba-4757-8264-01bf9d03db99-kube-api-access-k5vns\" (UniqueName: \"kubernetes.io/projected/98a11942-14ba-4757-8264-01bf9d03db99-kube-api-access-k5vns\") pod \"98a11942-14ba-4757-8264-01bf9d03db99\" (UID: \"98a11942-14ba-4757-8264-01bf9d03db99\") "
Oct 30 23:23:27.698833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:27.698787    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98a11942-14ba-4757-8264-01bf9d03db99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a11942-14ba-4757-8264-01bf9d03db99-maas-api-xks-ca\") pod \"98a11942-14ba-4757-8264-01bf9d03db99\" (UID: \"98a11942-14ba-4757-8264-01bf9d03db99\") "
Oct 30 23:23:27.698964 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:27.698937    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a11942-14ba-4757-8264-01bf9d03db99-maas-api-openshift-ca" pod "98a11942-14ba-4757-8264-01bf9d03db99" (UID: "98a11942-14ba-4757-8264-01bf9d03db99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:23:27.699082 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:27.699064    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98a11942-14ba-4757-8264-01bf9d03db99-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:27.700984 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:27.700961    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98a11942-14ba-4757-8264-01bf9d03db99-maas-api-xks-ca" pod "98a11942-14ba-4757-8264-01bf9d03db99" (UID: "98a11942-14ba-4757-8264-01bf9d03db99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:23:27.701081 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:27.701017    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a11942-14ba-4757-8264-01bf9d03db99-kube-api-access-k5vns" pod "98a11942-14ba-4757-8264-01bf9d03db99" (UID: "98a11942-14ba-4757-8264-01bf9d03db99"). InnerVolumeSpecName "kube-api-access-k5vns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:23:27.800021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:27.799993    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5vns\" (UniqueName: \"kubernetes.io/projected/98a11942-14ba-4757-8264-01bf9d03db99-kube-api-access-k5vns\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:27.800021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:27.800018    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98a11942-14ba-4757-8264-01bf9d03db99-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:28.478624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:28.478585    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m"
Oct 30 23:23:28.478624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:28.478587    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m" event={"ID":"98a11942-14ba-4757-8264-01bf9d03db99","Type":"ContainerDied","Data":"1d30a60e62e436471bdf3843132bb50b598f012a052c972676f55aeecd90aebe"}
Oct 30 23:23:28.478809 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:28.478652    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d30a60e62e436471bdf3843132bb50b598f012a052c972676f55aeecd90aebe"
Oct 30 23:23:37.577489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.577412    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs"]
Oct 30 23:23:37.577984 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.577847    5078 cpu_manager.go:414] "containerMap: removing container" podUID="98a11942-14ba-4757-8264-01bf9d03db99" containerName="revoke-keys"
Oct 30 23:23:37.577984 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.577859    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a11942-14ba-4757-8264-01bf9d03db99" containerName="revoke-keys"
Oct 30 23:23:37.577984 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.577936    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a11942-14ba-4757-8264-01bf9d03db99" containerName="revoke-keys"
Oct 30 23:23:37.580891 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.580874    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs"
Oct 30 23:23:37.634384 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.634362    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:23:37.683288 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.683259    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:23:37.683420 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.683289    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:23:37.683420 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.683403    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:23:37.685662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.685646    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs"]
Oct 30 23:23:37.701368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.701345    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ee03898-z8cfs\" (UID: \"3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs"
Oct 30 23:23:37.701512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.701494    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65kf9\" (UniqueName: \"kubernetes.io/projected/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-kube-api-access-65kf9\") pod \"maas-api-revoke-sub-3ee03898-z8cfs\" (UID: \"3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs"
Oct 30 23:23:37.701584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.701530    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ee03898-z8cfs\" (UID: \"3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs"
Oct 30 23:23:37.802734 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.802700    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc"]
Oct 30 23:23:37.802907 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.802813    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65kf9\" (UniqueName: \"kubernetes.io/projected/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-kube-api-access-65kf9\") pod \"maas-api-revoke-sub-3ee03898-z8cfs\" (UID: \"3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs"
Oct 30 23:23:37.802907 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.802852    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ee03898-z8cfs\" (UID: \"3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs"
Oct 30 23:23:37.803019 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.802942    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ee03898-z8cfs\" (UID: \"3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs"
Oct 30 23:23:37.803561 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.803542    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3ee03898-z8cfs\" (UID: \"3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs"
Oct 30 23:23:37.805654 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.805634    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3ee03898-z8cfs\" (UID: \"3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs"
Oct 30 23:23:37.806714 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.806701    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc"
Oct 30 23:23:37.859737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.859539    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65kf9\" (UniqueName: \"kubernetes.io/projected/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-kube-api-access-65kf9\") pod \"maas-api-revoke-sub-3ee03898-z8cfs\" (UID: \"3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs"
Oct 30 23:23:37.863899 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.863873    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc"]
Oct 30 23:23:37.893879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.893853    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs"
Oct 30 23:23:37.904226 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.904196    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tzsm\" (UniqueName: \"kubernetes.io/projected/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-kube-api-access-2tzsm\") pod \"maas-api-revoke-sub-f988c687-bbtvc\" (UID: \"d2b5a4f8-65b5-4d66-8921-521ff24d61e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc"
Oct 30 23:23:37.904348 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.904258    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f988c687-bbtvc\" (UID: \"d2b5a4f8-65b5-4d66-8921-521ff24d61e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc"
Oct 30 23:23:37.904412 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:37.904389    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f988c687-bbtvc\" (UID: \"d2b5a4f8-65b5-4d66-8921-521ff24d61e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc"
Oct 30 23:23:38.005962 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:38.005935    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f988c687-bbtvc\" (UID: \"d2b5a4f8-65b5-4d66-8921-521ff24d61e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc"
Oct 30 23:23:38.006119 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:38.006018    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tzsm\" (UniqueName: \"kubernetes.io/projected/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-kube-api-access-2tzsm\") pod \"maas-api-revoke-sub-f988c687-bbtvc\" (UID: \"d2b5a4f8-65b5-4d66-8921-521ff24d61e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc"
Oct 30 23:23:38.006119 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:38.006066    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f988c687-bbtvc\" (UID: \"d2b5a4f8-65b5-4d66-8921-521ff24d61e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc"
Oct 30 23:23:38.006760 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:38.006652    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f988c687-bbtvc\" (UID: \"d2b5a4f8-65b5-4d66-8921-521ff24d61e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc"
Oct 30 23:23:38.008353 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:38.008330    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f988c687-bbtvc\" (UID: \"d2b5a4f8-65b5-4d66-8921-521ff24d61e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc"
Oct 30 23:23:38.062568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:38.062545    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tzsm\" (UniqueName: \"kubernetes.io/projected/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-kube-api-access-2tzsm\") pod \"maas-api-revoke-sub-f988c687-bbtvc\" (UID: \"d2b5a4f8-65b5-4d66-8921-521ff24d61e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc"
Oct 30 23:23:38.128191 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:38.128131    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc"
Oct 30 23:23:38.281812 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:38.281762    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs"]
Oct 30 23:23:38.286288 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:23:38.286250    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e4ccf4d_8f88_42d6_bc04_fdc7c5cfade3.slice/crio-42f4f502fc5a46cb9ae4216dac108156d585ca047782c3a3a3f34d4b6f59cc8b WatchSource:0}: Error finding container 42f4f502fc5a46cb9ae4216dac108156d585ca047782c3a3a3f34d4b6f59cc8b: Status 404 returned error can't find the container with id 42f4f502fc5a46cb9ae4216dac108156d585ca047782c3a3a3f34d4b6f59cc8b
Oct 30 23:23:38.499909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:38.499887    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc"]
Oct 30 23:23:38.501370 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:23:38.501341    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b5a4f8_65b5_4d66_8921_521ff24d61e1.slice/crio-6aea0c4e6e353b7f1eff5493130d4031fdfe747d2ba3394dba59296895a960e7 WatchSource:0}: Error finding container 6aea0c4e6e353b7f1eff5493130d4031fdfe747d2ba3394dba59296895a960e7: Status 404 returned error can't find the container with id 6aea0c4e6e353b7f1eff5493130d4031fdfe747d2ba3394dba59296895a960e7
Oct 30 23:23:38.533577 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:38.533550    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs" event={"ID":"3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3","Type":"ContainerStarted","Data":"42f4f502fc5a46cb9ae4216dac108156d585ca047782c3a3a3f34d4b6f59cc8b"}
Oct 30 23:23:38.534701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:38.534666    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc" event={"ID":"d2b5a4f8-65b5-4d66-8921-521ff24d61e1","Type":"ContainerStarted","Data":"6aea0c4e6e353b7f1eff5493130d4031fdfe747d2ba3394dba59296895a960e7"}
Oct 30 23:23:39.541232 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:39.541201    5078 generic.go:356] "Generic (PLEG): container finished" podID="3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3" containerID="1ef07d85670047e01d4fb1b969141dcbffab8b2b7c71fad378f19e446ac165f7" exitCode=0
Oct 30 23:23:39.541643 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:39.541285    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs" event={"ID":"3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3","Type":"ContainerDied","Data":"1ef07d85670047e01d4fb1b969141dcbffab8b2b7c71fad378f19e446ac165f7"}
Oct 30 23:23:39.542949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:39.542925    5078 generic.go:356] "Generic (PLEG): container finished" podID="d2b5a4f8-65b5-4d66-8921-521ff24d61e1" containerID="5aed674adec47f6a26a98f31be24c4fd38b8e15b84ffa4455374785392d1d3a5" exitCode=0
Oct 30 23:23:39.543064 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:39.542999    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc" event={"ID":"d2b5a4f8-65b5-4d66-8921-521ff24d61e1","Type":"ContainerDied","Data":"5aed674adec47f6a26a98f31be24c4fd38b8e15b84ffa4455374785392d1d3a5"}
Oct 30 23:23:40.687616 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.687568    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc"
Oct 30 23:23:40.729065 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.729044    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs"
Oct 30 23:23:40.833473 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.833389    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-maas-api-openshift-ca\") pod \"3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3\" (UID: \"3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3\") "
Oct 30 23:23:40.833473 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.833442    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-maas-api-openshift-ca\") pod \"d2b5a4f8-65b5-4d66-8921-521ff24d61e1\" (UID: \"d2b5a4f8-65b5-4d66-8921-521ff24d61e1\") "
Oct 30 23:23:40.833734 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.833485    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-kube-api-access-65kf9\" (UniqueName: \"kubernetes.io/projected/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-kube-api-access-65kf9\") pod \"3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3\" (UID: \"3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3\") "
Oct 30 23:23:40.833734 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.833562    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-kube-api-access-2tzsm\" (UniqueName: \"kubernetes.io/projected/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-kube-api-access-2tzsm\") pod \"d2b5a4f8-65b5-4d66-8921-521ff24d61e1\" (UID: \"d2b5a4f8-65b5-4d66-8921-521ff24d61e1\") "
Oct 30 23:23:40.833734 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.833671    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-maas-api-xks-ca\") pod \"3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3\" (UID: \"3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3\") "
Oct 30 23:23:40.833734 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.833714    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-maas-api-xks-ca\") pod \"d2b5a4f8-65b5-4d66-8921-521ff24d61e1\" (UID: \"d2b5a4f8-65b5-4d66-8921-521ff24d61e1\") "
Oct 30 23:23:40.834056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.833794    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-maas-api-openshift-ca" pod "3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3" (UID: "3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:23:40.834156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.833840    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-maas-api-openshift-ca" pod "d2b5a4f8-65b5-4d66-8921-521ff24d61e1" (UID: "d2b5a4f8-65b5-4d66-8921-521ff24d61e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:23:40.834394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.834373    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:40.834463 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.834402    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:40.835964 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.835935    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-kube-api-access-65kf9" pod "3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3" (UID: "3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3"). InnerVolumeSpecName "kube-api-access-65kf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:23:40.836390 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.836361    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-maas-api-xks-ca" pod "d2b5a4f8-65b5-4d66-8921-521ff24d61e1" (UID: "d2b5a4f8-65b5-4d66-8921-521ff24d61e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:23:40.836493 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.836427    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-maas-api-xks-ca" pod "3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3" (UID: "3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:23:40.836539 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.836505    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-kube-api-access-2tzsm" pod "d2b5a4f8-65b5-4d66-8921-521ff24d61e1" (UID: "d2b5a4f8-65b5-4d66-8921-521ff24d61e1"). InnerVolumeSpecName "kube-api-access-2tzsm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:23:40.936039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.936006    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:40.936039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.936040    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:40.936219 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.936057    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65kf9\" (UniqueName: \"kubernetes.io/projected/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3-kube-api-access-65kf9\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:40.936219 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:40.936073    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tzsm\" (UniqueName: \"kubernetes.io/projected/d2b5a4f8-65b5-4d66-8921-521ff24d61e1-kube-api-access-2tzsm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:41.555235 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:41.555207    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc"
Oct 30 23:23:41.555408 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:41.555247    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc" event={"ID":"d2b5a4f8-65b5-4d66-8921-521ff24d61e1","Type":"ContainerDied","Data":"6aea0c4e6e353b7f1eff5493130d4031fdfe747d2ba3394dba59296895a960e7"}
Oct 30 23:23:41.555408 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:41.555292    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6aea0c4e6e353b7f1eff5493130d4031fdfe747d2ba3394dba59296895a960e7"
Oct 30 23:23:41.564057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:41.564025    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs" event={"ID":"3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3","Type":"ContainerDied","Data":"42f4f502fc5a46cb9ae4216dac108156d585ca047782c3a3a3f34d4b6f59cc8b"}
Oct 30 23:23:41.564205 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:41.564095    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42f4f502fc5a46cb9ae4216dac108156d585ca047782c3a3a3f34d4b6f59cc8b"
Oct 30 23:23:41.564205 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:41.564035    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs"
Oct 30 23:23:45.265495 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.265455    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8"]
Oct 30 23:23:45.265906 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.265879    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d2b5a4f8-65b5-4d66-8921-521ff24d61e1" containerName="revoke-keys"
Oct 30 23:23:45.265906 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.265890    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b5a4f8-65b5-4d66-8921-521ff24d61e1" containerName="revoke-keys"
Oct 30 23:23:45.265986 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.265921    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3" containerName="revoke-keys"
Oct 30 23:23:45.265986 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.265926    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3" containerName="revoke-keys"
Oct 30 23:23:45.266042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.266000    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b5a4f8-65b5-4d66-8921-521ff24d61e1" containerName="revoke-keys"
Oct 30 23:23:45.266042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.266011    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3" containerName="revoke-keys"
Oct 30 23:23:45.269291 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.269275    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8"
Oct 30 23:23:45.325487 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.325450    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:23:45.374931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.374885    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc745fc-5bb4-4a2d-af49-02c5da152d46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9b87a6a-n5pt8\" (UID: \"3bc745fc-5bb4-4a2d-af49-02c5da152d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8"
Oct 30 23:23:45.375105 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.375012    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc745fc-5bb4-4a2d-af49-02c5da152d46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9b87a6a-n5pt8\" (UID: \"3bc745fc-5bb4-4a2d-af49-02c5da152d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8"
Oct 30 23:23:45.375105 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.375072    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-css4n\" (UniqueName: \"kubernetes.io/projected/3bc745fc-5bb4-4a2d-af49-02c5da152d46-kube-api-access-css4n\") pod \"maas-api-revoke-sub-f9b87a6a-n5pt8\" (UID: \"3bc745fc-5bb4-4a2d-af49-02c5da152d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8"
Oct 30 23:23:45.375250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.375215    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:23:45.375307 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.375278    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:23:45.375373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.375357    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:23:45.377667 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.377645    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8"]
Oct 30 23:23:45.476474 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.476446    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-css4n\" (UniqueName: \"kubernetes.io/projected/3bc745fc-5bb4-4a2d-af49-02c5da152d46-kube-api-access-css4n\") pod \"maas-api-revoke-sub-f9b87a6a-n5pt8\" (UID: \"3bc745fc-5bb4-4a2d-af49-02c5da152d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8"
Oct 30 23:23:45.476664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.476550    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc745fc-5bb4-4a2d-af49-02c5da152d46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9b87a6a-n5pt8\" (UID: \"3bc745fc-5bb4-4a2d-af49-02c5da152d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8"
Oct 30 23:23:45.476664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.476621    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc745fc-5bb4-4a2d-af49-02c5da152d46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9b87a6a-n5pt8\" (UID: \"3bc745fc-5bb4-4a2d-af49-02c5da152d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8"
Oct 30 23:23:45.477325 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.477296    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc745fc-5bb4-4a2d-af49-02c5da152d46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9b87a6a-n5pt8\" (UID: \"3bc745fc-5bb4-4a2d-af49-02c5da152d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8"
Oct 30 23:23:45.478996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.478978    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc745fc-5bb4-4a2d-af49-02c5da152d46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9b87a6a-n5pt8\" (UID: \"3bc745fc-5bb4-4a2d-af49-02c5da152d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8"
Oct 30 23:23:45.535103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.535031    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-css4n\" (UniqueName: \"kubernetes.io/projected/3bc745fc-5bb4-4a2d-af49-02c5da152d46-kube-api-access-css4n\") pod \"maas-api-revoke-sub-f9b87a6a-n5pt8\" (UID: \"3bc745fc-5bb4-4a2d-af49-02c5da152d46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8"
Oct 30 23:23:45.582652 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.582624    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8"
Oct 30 23:23:45.955129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:45.955097    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8"]
Oct 30 23:23:45.956583 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:23:45.956560    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc745fc_5bb4_4a2d_af49_02c5da152d46.slice/crio-08f8eb95d4d24c1e4708a8f25a2ac37a8bc95af93734219fa38ddae21001e434 WatchSource:0}: Error finding container 08f8eb95d4d24c1e4708a8f25a2ac37a8bc95af93734219fa38ddae21001e434: Status 404 returned error can't find the container with id 08f8eb95d4d24c1e4708a8f25a2ac37a8bc95af93734219fa38ddae21001e434
Oct 30 23:23:46.590047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:46.590015    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8" event={"ID":"3bc745fc-5bb4-4a2d-af49-02c5da152d46","Type":"ContainerStarted","Data":"4c66a6c566dfdce3fe349746466374faf2d95e8b98e1556e942158c55fe554ce"}
Oct 30 23:23:46.590431 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:46.590067    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8" event={"ID":"3bc745fc-5bb4-4a2d-af49-02c5da152d46","Type":"ContainerStarted","Data":"08f8eb95d4d24c1e4708a8f25a2ac37a8bc95af93734219fa38ddae21001e434"}
Oct 30 23:23:46.704475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:46.704424    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8" podStartSLOduration=1.70440795 podStartE2EDuration="1.70440795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:23:46.702836475 +0000 UTC m=+1012.590742829" watchObservedRunningTime="2026-10-30 23:23:46.70440795 +0000 UTC m=+1012.592314307"
Oct 30 23:23:47.590735 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:23:47.590694    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:23:47.598024 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:47.597994    5078 generic.go:356] "Generic (PLEG): container finished" podID="3bc745fc-5bb4-4a2d-af49-02c5da152d46" containerID="4c66a6c566dfdce3fe349746466374faf2d95e8b98e1556e942158c55fe554ce" exitCode=0
Oct 30 23:23:47.598175 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:47.598080    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8" event={"ID":"3bc745fc-5bb4-4a2d-af49-02c5da152d46","Type":"ContainerDied","Data":"4c66a6c566dfdce3fe349746466374faf2d95e8b98e1556e942158c55fe554ce"}
Oct 30 23:23:48.748916 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:48.748889    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8"
Oct 30 23:23:48.812044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:48.812019    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc745fc-5bb4-4a2d-af49-02c5da152d46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc745fc-5bb4-4a2d-af49-02c5da152d46-maas-api-openshift-ca\") pod \"3bc745fc-5bb4-4a2d-af49-02c5da152d46\" (UID: \"3bc745fc-5bb4-4a2d-af49-02c5da152d46\") "
Oct 30 23:23:48.812178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:48.812052    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bc745fc-5bb4-4a2d-af49-02c5da152d46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc745fc-5bb4-4a2d-af49-02c5da152d46-maas-api-xks-ca\") pod \"3bc745fc-5bb4-4a2d-af49-02c5da152d46\" (UID: \"3bc745fc-5bb4-4a2d-af49-02c5da152d46\") "
Oct 30 23:23:48.812178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:48.812157    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc745fc-5bb4-4a2d-af49-02c5da152d46-kube-api-access-css4n\" (UniqueName: \"kubernetes.io/projected/3bc745fc-5bb4-4a2d-af49-02c5da152d46-kube-api-access-css4n\") pod \"3bc745fc-5bb4-4a2d-af49-02c5da152d46\" (UID: \"3bc745fc-5bb4-4a2d-af49-02c5da152d46\") "
Oct 30 23:23:48.812354 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:48.812330    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc745fc-5bb4-4a2d-af49-02c5da152d46-maas-api-openshift-ca" pod "3bc745fc-5bb4-4a2d-af49-02c5da152d46" (UID: "3bc745fc-5bb4-4a2d-af49-02c5da152d46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:23:48.814342 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:48.814315    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc745fc-5bb4-4a2d-af49-02c5da152d46-maas-api-xks-ca" pod "3bc745fc-5bb4-4a2d-af49-02c5da152d46" (UID: "3bc745fc-5bb4-4a2d-af49-02c5da152d46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:23:48.814444 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:48.814364    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc745fc-5bb4-4a2d-af49-02c5da152d46-kube-api-access-css4n" pod "3bc745fc-5bb4-4a2d-af49-02c5da152d46" (UID: "3bc745fc-5bb4-4a2d-af49-02c5da152d46"). InnerVolumeSpecName "kube-api-access-css4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:23:48.913655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:48.913574    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc745fc-5bb4-4a2d-af49-02c5da152d46-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:48.913655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:48.913616    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc745fc-5bb4-4a2d-af49-02c5da152d46-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:48.913655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:48.913631    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-css4n\" (UniqueName: \"kubernetes.io/projected/3bc745fc-5bb4-4a2d-af49-02c5da152d46-kube-api-access-css4n\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:49.617138 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:49.617106    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8"
Oct 30 23:23:49.617332 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:49.617147    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8" event={"ID":"3bc745fc-5bb4-4a2d-af49-02c5da152d46","Type":"ContainerDied","Data":"08f8eb95d4d24c1e4708a8f25a2ac37a8bc95af93734219fa38ddae21001e434"}
Oct 30 23:23:49.617332 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:49.617229    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08f8eb95d4d24c1e4708a8f25a2ac37a8bc95af93734219fa38ddae21001e434"
Oct 30 23:23:52.711527 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.711492    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"]
Oct 30 23:23:52.711962 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.711948    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3bc745fc-5bb4-4a2d-af49-02c5da152d46" containerName="revoke-keys"
Oct 30 23:23:52.712001 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.711963    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc745fc-5bb4-4a2d-af49-02c5da152d46" containerName="revoke-keys"
Oct 30 23:23:52.712054 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.712044    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc745fc-5bb4-4a2d-af49-02c5da152d46" containerName="revoke-keys"
Oct 30 23:23:52.715467 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.715449    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.754785 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.754759    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a87c532-4c80-46b8-98ee-4b07dadba3b2-tls-certs\") pod \"distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") " pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.754901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.754792    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-model-cache\") pod \"distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") " pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.754901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.754824    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-home\") pod \"distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") " pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.755004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.754970    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-dshm\") pod \"distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") " pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.755050 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.755013    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-tmp-dir\") pod \"distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") " pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.755107 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.755092    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5scf\" (UniqueName: \"kubernetes.io/projected/4a87c532-4c80-46b8-98ee-4b07dadba3b2-kube-api-access-k5scf\") pod \"distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") " pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.819140 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.819112    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-aadaa6\"/\"default-dockercfg-d4g98\""
Oct 30 23:23:52.819140 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.819132    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-aadaa6\"/\"kube-root-ca.crt\""
Oct 30 23:23:52.819313 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.819153    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-aadaa6\"/\"distinct-w6-aadaa6-kserve-self-signed-certs\""
Oct 30 23:23:52.819578 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.819560    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-aadaa6\"/\"openshift-service-ca.crt\""
Oct 30 23:23:52.821558 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.821542    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"]
Oct 30 23:23:52.878581 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.878536    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-home\") pod \"distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") " pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.878804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.878646    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-dshm\") pod \"distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") " pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.878804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.878678    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-tmp-dir\") pod \"distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") " pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.878804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.878731    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5scf\" (UniqueName: \"kubernetes.io/projected/4a87c532-4c80-46b8-98ee-4b07dadba3b2-kube-api-access-k5scf\") pod \"distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") " pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.878955 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.878869    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a87c532-4c80-46b8-98ee-4b07dadba3b2-tls-certs\") pod \"distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") " pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.878955 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.878897    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-model-cache\") pod \"distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") " pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.880645 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.879345    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-model-cache\") pod \"distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") " pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.880645 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.879625    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-home\") pod \"distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") " pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.882164 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.881800    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-tmp-dir\") pod \"distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") " pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.885291 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.885214    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-dshm\") pod \"distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") " pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.890560 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.889961    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a87c532-4c80-46b8-98ee-4b07dadba3b2-tls-certs\") pod \"distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") " pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.938461 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.938434    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5scf\" (UniqueName: \"kubernetes.io/projected/4a87c532-4c80-46b8-98ee-4b07dadba3b2-kube-api-access-k5scf\") pod \"distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") " pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:52.981096 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.981022    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz"]
Oct 30 23:23:52.985964 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:52.985941    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz"
Oct 30 23:23:53.027306 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.027287    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:53.038450 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.038429    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:23:53.038737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.038715    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:23:53.038858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.038775    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:23:53.039333 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.039311    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:23:53.042913 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.042887    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz"]
Oct 30 23:23:53.081590 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.081462    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c1fe84-7d11-49a3-b714-198724453ebc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a7d52ee-clhvz\" (UID: \"74c1fe84-7d11-49a3-b714-198724453ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz"
Oct 30 23:23:53.082226 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.081869    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c1fe84-7d11-49a3-b714-198724453ebc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a7d52ee-clhvz\" (UID: \"74c1fe84-7d11-49a3-b714-198724453ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz"
Oct 30 23:23:53.082226 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.082043    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfcjc\" (UniqueName: \"kubernetes.io/projected/74c1fe84-7d11-49a3-b714-198724453ebc-kube-api-access-gfcjc\") pod \"maas-api-revoke-sub-8a7d52ee-clhvz\" (UID: \"74c1fe84-7d11-49a3-b714-198724453ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz"
Oct 30 23:23:53.183644 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.183586    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfcjc\" (UniqueName: \"kubernetes.io/projected/74c1fe84-7d11-49a3-b714-198724453ebc-kube-api-access-gfcjc\") pod \"maas-api-revoke-sub-8a7d52ee-clhvz\" (UID: \"74c1fe84-7d11-49a3-b714-198724453ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz"
Oct 30 23:23:53.183817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.183734    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c1fe84-7d11-49a3-b714-198724453ebc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a7d52ee-clhvz\" (UID: \"74c1fe84-7d11-49a3-b714-198724453ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz"
Oct 30 23:23:53.183817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.183807    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c1fe84-7d11-49a3-b714-198724453ebc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a7d52ee-clhvz\" (UID: \"74c1fe84-7d11-49a3-b714-198724453ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz"
Oct 30 23:23:53.184490 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.184461    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c1fe84-7d11-49a3-b714-198724453ebc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8a7d52ee-clhvz\" (UID: \"74c1fe84-7d11-49a3-b714-198724453ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz"
Oct 30 23:23:53.187214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.187190    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c1fe84-7d11-49a3-b714-198724453ebc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8a7d52ee-clhvz\" (UID: \"74c1fe84-7d11-49a3-b714-198724453ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz"
Oct 30 23:23:53.240448 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.240386    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfcjc\" (UniqueName: \"kubernetes.io/projected/74c1fe84-7d11-49a3-b714-198724453ebc-kube-api-access-gfcjc\") pod \"maas-api-revoke-sub-8a7d52ee-clhvz\" (UID: \"74c1fe84-7d11-49a3-b714-198724453ebc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz"
Oct 30 23:23:53.299480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.299458    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz"
Oct 30 23:23:53.416029 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.413882    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"]
Oct 30 23:23:53.639904 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.639873    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" event={"ID":"4a87c532-4c80-46b8-98ee-4b07dadba3b2","Type":"ContainerStarted","Data":"b3df52657d928af74af45a7d355b3d186621068ddf0f0caa321bf87a8c704817"}
Oct 30 23:23:53.640085 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.639912    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" event={"ID":"4a87c532-4c80-46b8-98ee-4b07dadba3b2","Type":"ContainerStarted","Data":"7004d6a3c0d878959d4449389ea4f6fe8dc88f580c964c5eab4a40d904f38d52"}
Oct 30 23:23:53.669970 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.669948    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz"]
Oct 30 23:23:53.670718 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:23:53.670696    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c1fe84_7d11_49a3_b714_198724453ebc.slice/crio-eb9a42e0bfe31e2435f2571b47748450240a4a88d9c3438ec2b8e868765e366b WatchSource:0}: Error finding container eb9a42e0bfe31e2435f2571b47748450240a4a88d9c3438ec2b8e868765e366b: Status 404 returned error can't find the container with id eb9a42e0bfe31e2435f2571b47748450240a4a88d9c3438ec2b8e868765e366b
Oct 30 23:23:53.754980 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.754936    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podStartSLOduration=1.754922638 podStartE2EDuration="1.754922638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:23:53.753289583 +0000 UTC m=+1019.641195962" watchObservedRunningTime="2026-10-30 23:23:53.754922638 +0000 UTC m=+1019.642828992"
Oct 30 23:23:53.882620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.881574    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv"]
Oct 30 23:23:53.892985 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.892906    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv"
Oct 30 23:23:53.939513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.939484    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv"]
Oct 30 23:23:53.992075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.992052    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2278116c-cd84-4312-999a-832dae090303-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecf73501-gfxkv\" (UID: \"2278116c-cd84-4312-999a-832dae090303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv"
Oct 30 23:23:53.992178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.992082    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2278116c-cd84-4312-999a-832dae090303-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecf73501-gfxkv\" (UID: \"2278116c-cd84-4312-999a-832dae090303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv"
Oct 30 23:23:53.992178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:53.992109    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2whjl\" (UniqueName: \"kubernetes.io/projected/2278116c-cd84-4312-999a-832dae090303-kube-api-access-2whjl\") pod \"maas-api-revoke-sub-ecf73501-gfxkv\" (UID: \"2278116c-cd84-4312-999a-832dae090303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv"
Oct 30 23:23:54.027745 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.027722    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:23:54.093503 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.093472    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2278116c-cd84-4312-999a-832dae090303-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecf73501-gfxkv\" (UID: \"2278116c-cd84-4312-999a-832dae090303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv"
Oct 30 23:23:54.093681 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.093507    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2278116c-cd84-4312-999a-832dae090303-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecf73501-gfxkv\" (UID: \"2278116c-cd84-4312-999a-832dae090303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv"
Oct 30 23:23:54.093681 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.093535    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2whjl\" (UniqueName: \"kubernetes.io/projected/2278116c-cd84-4312-999a-832dae090303-kube-api-access-2whjl\") pod \"maas-api-revoke-sub-ecf73501-gfxkv\" (UID: \"2278116c-cd84-4312-999a-832dae090303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv"
Oct 30 23:23:54.094167 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.094141    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2278116c-cd84-4312-999a-832dae090303-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ecf73501-gfxkv\" (UID: \"2278116c-cd84-4312-999a-832dae090303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv"
Oct 30 23:23:54.095961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.095943    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2278116c-cd84-4312-999a-832dae090303-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ecf73501-gfxkv\" (UID: \"2278116c-cd84-4312-999a-832dae090303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv"
Oct 30 23:23:54.151136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.151082    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2whjl\" (UniqueName: \"kubernetes.io/projected/2278116c-cd84-4312-999a-832dae090303-kube-api-access-2whjl\") pod \"maas-api-revoke-sub-ecf73501-gfxkv\" (UID: \"2278116c-cd84-4312-999a-832dae090303\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv"
Oct 30 23:23:54.214339 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.214313    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv"
Oct 30 23:23:54.584619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.584578    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv"]
Oct 30 23:23:54.594519 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:23:54.594493    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2278116c_cd84_4312_999a_832dae090303.slice/crio-98d6be47b1f5e3dd465090e893c8882eaeb3baa65e888c711f3986940af20565 WatchSource:0}: Error finding container 98d6be47b1f5e3dd465090e893c8882eaeb3baa65e888c711f3986940af20565: Status 404 returned error can't find the container with id 98d6be47b1f5e3dd465090e893c8882eaeb3baa65e888c711f3986940af20565
Oct 30 23:23:54.646437 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.646411    5078 generic.go:356] "Generic (PLEG): container finished" podID="74c1fe84-7d11-49a3-b714-198724453ebc" containerID="272eecaee3983794b7a094a58beb98ef42bb4f6c8ceabe5400d5dabc6bd147b3" exitCode=0
Oct 30 23:23:54.646543 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.646493    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz" event={"ID":"74c1fe84-7d11-49a3-b714-198724453ebc","Type":"ContainerDied","Data":"272eecaee3983794b7a094a58beb98ef42bb4f6c8ceabe5400d5dabc6bd147b3"}
Oct 30 23:23:54.646584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.646554    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz" event={"ID":"74c1fe84-7d11-49a3-b714-198724453ebc","Type":"ContainerStarted","Data":"eb9a42e0bfe31e2435f2571b47748450240a4a88d9c3438ec2b8e868765e366b"}
Oct 30 23:23:54.647825 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.647806    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv" event={"ID":"2278116c-cd84-4312-999a-832dae090303","Type":"ContainerStarted","Data":"98d6be47b1f5e3dd465090e893c8882eaeb3baa65e888c711f3986940af20565"}
Oct 30 23:23:54.746296 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.746267    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"]
Oct 30 23:23:54.750954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.750936    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:54.845222 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.845194    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"]
Oct 30 23:23:54.900882 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.900857    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4"]
Oct 30 23:23:54.901296 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.901277    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:54.901363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.901322    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:54.901415 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.901386    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz9vn\" (UniqueName: \"kubernetes.io/projected/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-kube-api-access-vz9vn\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:54.901480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.901462    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:54.901537 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.901513    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:54.901583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.901557    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:54.901719 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.901700    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:54.901779 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.901731    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:54.901779 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.901759    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:54.904392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.904379    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4"
Oct 30 23:23:54.964772 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:54.964744    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4"]
Oct 30 23:23:55.002920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.002889    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz9vn\" (UniqueName: \"kubernetes.io/projected/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-kube-api-access-vz9vn\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.003045 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.002936    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.003086 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.003051    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.003131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.003106    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.003174 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.003150    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d23c612f-7a03-4b86-af74-6315ddf4657a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-109eace1-w9df4\" (UID: \"d23c612f-7a03-4b86-af74-6315ddf4657a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4"
Oct 30 23:23:55.003211 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.003184    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d23c612f-7a03-4b86-af74-6315ddf4657a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-109eace1-w9df4\" (UID: \"d23c612f-7a03-4b86-af74-6315ddf4657a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4"
Oct 30 23:23:55.003327 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.003313    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.003386 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.003341    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.003386 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.003366    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6r5v\" (UniqueName: \"kubernetes.io/projected/d23c612f-7a03-4b86-af74-6315ddf4657a-kube-api-access-k6r5v\") pod \"maas-api-revoke-sub-109eace1-w9df4\" (UID: \"d23c612f-7a03-4b86-af74-6315ddf4657a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4"
Oct 30 23:23:55.003488 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.003397    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.003488 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.003453    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.003586 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.003504    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.003737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.003709    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.003737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.003729    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.004064 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.003750    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.004064 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.003881    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.004064 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.003945    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.005728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.005710    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.005799 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.005729    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.060400 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.060375    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.060689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.060667    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz9vn\" (UniqueName: \"kubernetes.io/projected/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-kube-api-access-vz9vn\") pod \"maas-default-gateway-openshift-default-5df48fcc99-8nrmp\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.065283 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.065267    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:55.104816 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.104787    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6r5v\" (UniqueName: \"kubernetes.io/projected/d23c612f-7a03-4b86-af74-6315ddf4657a-kube-api-access-k6r5v\") pod \"maas-api-revoke-sub-109eace1-w9df4\" (UID: \"d23c612f-7a03-4b86-af74-6315ddf4657a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4"
Oct 30 23:23:55.104963 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.104913    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d23c612f-7a03-4b86-af74-6315ddf4657a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-109eace1-w9df4\" (UID: \"d23c612f-7a03-4b86-af74-6315ddf4657a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4"
Oct 30 23:23:55.104963 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.104948    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d23c612f-7a03-4b86-af74-6315ddf4657a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-109eace1-w9df4\" (UID: \"d23c612f-7a03-4b86-af74-6315ddf4657a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4"
Oct 30 23:23:55.106382 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.105695    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d23c612f-7a03-4b86-af74-6315ddf4657a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-109eace1-w9df4\" (UID: \"d23c612f-7a03-4b86-af74-6315ddf4657a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4"
Oct 30 23:23:55.107833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.107812    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d23c612f-7a03-4b86-af74-6315ddf4657a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-109eace1-w9df4\" (UID: \"d23c612f-7a03-4b86-af74-6315ddf4657a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4"
Oct 30 23:23:55.204818 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.204753    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6r5v\" (UniqueName: \"kubernetes.io/projected/d23c612f-7a03-4b86-af74-6315ddf4657a-kube-api-access-k6r5v\") pod \"maas-api-revoke-sub-109eace1-w9df4\" (UID: \"d23c612f-7a03-4b86-af74-6315ddf4657a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4"
Oct 30 23:23:55.219800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.218624    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4"
Oct 30 23:23:55.497355 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.497325    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"]
Oct 30 23:23:55.498215 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:23:55.498182    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84900961_7d7e_4bf0_9c40_6f3d0d8102a1.slice/crio-abc33f450293e953b8c6aabc0f3de8f0e59f7bd96fb8b361f80690b7b3e9cef1 WatchSource:0}: Error finding container abc33f450293e953b8c6aabc0f3de8f0e59f7bd96fb8b361f80690b7b3e9cef1: Status 404 returned error can't find the container with id abc33f450293e953b8c6aabc0f3de8f0e59f7bd96fb8b361f80690b7b3e9cef1
Oct 30 23:23:55.500971 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.500932    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:23:55.501039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.501021    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:23:55.501075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.501064    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:23:55.615008 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:23:55.614715    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23c612f_7a03_4b86_af74_6315ddf4657a.slice/crio-b8a94d3303ca394046437e135a2c523dd22736f7c3844c6b0ca29685817942f1 WatchSource:0}: Error finding container b8a94d3303ca394046437e135a2c523dd22736f7c3844c6b0ca29685817942f1: Status 404 returned error can't find the container with id b8a94d3303ca394046437e135a2c523dd22736f7c3844c6b0ca29685817942f1
Oct 30 23:23:55.619932 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.618990    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4"]
Oct 30 23:23:55.658427 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.658395    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4" event={"ID":"d23c612f-7a03-4b86-af74-6315ddf4657a","Type":"ContainerStarted","Data":"b8a94d3303ca394046437e135a2c523dd22736f7c3844c6b0ca29685817942f1"}
Oct 30 23:23:55.660338 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.660316    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp" event={"ID":"84900961-7d7e-4bf0-9c40-6f3d0d8102a1","Type":"ContainerStarted","Data":"b89fe26163e55f5a2f54ec2c9934bdff2af9f6411da9f2916ae741e4bc33db41"}
Oct 30 23:23:55.660501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.660489    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp" event={"ID":"84900961-7d7e-4bf0-9c40-6f3d0d8102a1","Type":"ContainerStarted","Data":"abc33f450293e953b8c6aabc0f3de8f0e59f7bd96fb8b361f80690b7b3e9cef1"}
Oct 30 23:23:55.663664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.663642    5078 generic.go:356] "Generic (PLEG): container finished" podID="2278116c-cd84-4312-999a-832dae090303" containerID="9812c77e0524c27ebf2db069c9065bb8126b7b0f65c832233fe966611ad4f4cf" exitCode=0
Oct 30 23:23:55.664083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.664060    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv" event={"ID":"2278116c-cd84-4312-999a-832dae090303","Type":"ContainerDied","Data":"9812c77e0524c27ebf2db069c9065bb8126b7b0f65c832233fe966611ad4f4cf"}
Oct 30 23:23:55.808337 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:55.808264    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp" podStartSLOduration=1.8082420369999999 podStartE2EDuration="1.808242037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:23:55.779268103 +0000 UTC m=+1021.667174469" watchObservedRunningTime="2026-10-30 23:23:55.808242037 +0000 UTC m=+1021.696148387"
Oct 30 23:23:56.007550 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.007287    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz"
Oct 30 23:23:56.065768 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.065736    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:23:56.067740 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.067691    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8nrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:23:56.067876 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.067763    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp" podUID="84900961-7d7e-4bf0-9c40-6f3d0d8102a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 23:23:56.145784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.145750    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74c1fe84-7d11-49a3-b714-198724453ebc-kube-api-access-gfcjc\" (UniqueName: \"kubernetes.io/projected/74c1fe84-7d11-49a3-b714-198724453ebc-kube-api-access-gfcjc\") pod \"74c1fe84-7d11-49a3-b714-198724453ebc\" (UID: \"74c1fe84-7d11-49a3-b714-198724453ebc\") "
Oct 30 23:23:56.145967 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.145821    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74c1fe84-7d11-49a3-b714-198724453ebc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c1fe84-7d11-49a3-b714-198724453ebc-maas-api-xks-ca\") pod \"74c1fe84-7d11-49a3-b714-198724453ebc\" (UID: \"74c1fe84-7d11-49a3-b714-198724453ebc\") "
Oct 30 23:23:56.145967 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.145929    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74c1fe84-7d11-49a3-b714-198724453ebc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c1fe84-7d11-49a3-b714-198724453ebc-maas-api-openshift-ca\") pod \"74c1fe84-7d11-49a3-b714-198724453ebc\" (UID: \"74c1fe84-7d11-49a3-b714-198724453ebc\") "
Oct 30 23:23:56.146571 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.146529    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c1fe84-7d11-49a3-b714-198724453ebc-maas-api-openshift-ca" pod "74c1fe84-7d11-49a3-b714-198724453ebc" (UID: "74c1fe84-7d11-49a3-b714-198724453ebc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:23:56.148185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.148156    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c1fe84-7d11-49a3-b714-198724453ebc-kube-api-access-gfcjc" pod "74c1fe84-7d11-49a3-b714-198724453ebc" (UID: "74c1fe84-7d11-49a3-b714-198724453ebc"). InnerVolumeSpecName "kube-api-access-gfcjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:23:56.149189 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.149163    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74c1fe84-7d11-49a3-b714-198724453ebc-maas-api-xks-ca" pod "74c1fe84-7d11-49a3-b714-198724453ebc" (UID: "74c1fe84-7d11-49a3-b714-198724453ebc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:23:56.254811 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.253459    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfcjc\" (UniqueName: \"kubernetes.io/projected/74c1fe84-7d11-49a3-b714-198724453ebc-kube-api-access-gfcjc\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:56.254811 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.253497    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74c1fe84-7d11-49a3-b714-198724453ebc-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:56.254811 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.253613    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74c1fe84-7d11-49a3-b714-198724453ebc-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:56.689128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.689089    5078 generic.go:356] "Generic (PLEG): container finished" podID="d23c612f-7a03-4b86-af74-6315ddf4657a" containerID="4d536f1667d6314d2675a9c878bd2799b033063651d32b4df96d873fbdea6591" exitCode=0
Oct 30 23:23:56.689307 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.689169    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4" event={"ID":"d23c612f-7a03-4b86-af74-6315ddf4657a","Type":"ContainerDied","Data":"4d536f1667d6314d2675a9c878bd2799b033063651d32b4df96d873fbdea6591"}
Oct 30 23:23:56.690582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.690555    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz"
Oct 30 23:23:56.690582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.690582    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz" event={"ID":"74c1fe84-7d11-49a3-b714-198724453ebc","Type":"ContainerDied","Data":"eb9a42e0bfe31e2435f2571b47748450240a4a88d9c3438ec2b8e868765e366b"}
Oct 30 23:23:56.690771 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.690646    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb9a42e0bfe31e2435f2571b47748450240a4a88d9c3438ec2b8e868765e366b"
Oct 30 23:23:56.835351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.835329    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv"
Oct 30 23:23:56.990195 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.990165    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2278116c-cd84-4312-999a-832dae090303-kube-api-access-2whjl\" (UniqueName: \"kubernetes.io/projected/2278116c-cd84-4312-999a-832dae090303-kube-api-access-2whjl\") pod \"2278116c-cd84-4312-999a-832dae090303\" (UID: \"2278116c-cd84-4312-999a-832dae090303\") "
Oct 30 23:23:56.990368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.990214    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2278116c-cd84-4312-999a-832dae090303-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2278116c-cd84-4312-999a-832dae090303-maas-api-xks-ca\") pod \"2278116c-cd84-4312-999a-832dae090303\" (UID: \"2278116c-cd84-4312-999a-832dae090303\") "
Oct 30 23:23:56.990368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.990309    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2278116c-cd84-4312-999a-832dae090303-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2278116c-cd84-4312-999a-832dae090303-maas-api-openshift-ca\") pod \"2278116c-cd84-4312-999a-832dae090303\" (UID: \"2278116c-cd84-4312-999a-832dae090303\") "
Oct 30 23:23:56.990757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.990726    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2278116c-cd84-4312-999a-832dae090303-maas-api-openshift-ca" pod "2278116c-cd84-4312-999a-832dae090303" (UID: "2278116c-cd84-4312-999a-832dae090303"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:23:56.992728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.992696    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2278116c-cd84-4312-999a-832dae090303-kube-api-access-2whjl" pod "2278116c-cd84-4312-999a-832dae090303" (UID: "2278116c-cd84-4312-999a-832dae090303"). InnerVolumeSpecName "kube-api-access-2whjl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:23:56.993092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:56.993067    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2278116c-cd84-4312-999a-832dae090303-maas-api-xks-ca" pod "2278116c-cd84-4312-999a-832dae090303" (UID: "2278116c-cd84-4312-999a-832dae090303"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:23:57.081863 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:57.080202    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8nrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:23:57.081863 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:57.080263    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp" podUID="84900961-7d7e-4bf0-9c40-6f3d0d8102a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 23:23:57.091910 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:57.091881    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2278116c-cd84-4312-999a-832dae090303-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:57.091910 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:57.091906    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2whjl\" (UniqueName: \"kubernetes.io/projected/2278116c-cd84-4312-999a-832dae090303-kube-api-access-2whjl\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:57.092067 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:57.091918    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2278116c-cd84-4312-999a-832dae090303-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:57.717214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:57.717169    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv"
Oct 30 23:23:57.717214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:57.717201    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv" event={"ID":"2278116c-cd84-4312-999a-832dae090303","Type":"ContainerDied","Data":"98d6be47b1f5e3dd465090e893c8882eaeb3baa65e888c711f3986940af20565"}
Oct 30 23:23:57.717469 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:57.717288    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98d6be47b1f5e3dd465090e893c8882eaeb3baa65e888c711f3986940af20565"
Oct 30 23:23:57.889560 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:57.889533    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4"
Oct 30 23:23:58.009106 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:58.009032    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d23c612f-7a03-4b86-af74-6315ddf4657a-kube-api-access-k6r5v\" (UniqueName: \"kubernetes.io/projected/d23c612f-7a03-4b86-af74-6315ddf4657a-kube-api-access-k6r5v\") pod \"d23c612f-7a03-4b86-af74-6315ddf4657a\" (UID: \"d23c612f-7a03-4b86-af74-6315ddf4657a\") "
Oct 30 23:23:58.009274 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:58.009188    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d23c612f-7a03-4b86-af74-6315ddf4657a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d23c612f-7a03-4b86-af74-6315ddf4657a-maas-api-openshift-ca\") pod \"d23c612f-7a03-4b86-af74-6315ddf4657a\" (UID: \"d23c612f-7a03-4b86-af74-6315ddf4657a\") "
Oct 30 23:23:58.009274 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:58.009214    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d23c612f-7a03-4b86-af74-6315ddf4657a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d23c612f-7a03-4b86-af74-6315ddf4657a-maas-api-xks-ca\") pod \"d23c612f-7a03-4b86-af74-6315ddf4657a\" (UID: \"d23c612f-7a03-4b86-af74-6315ddf4657a\") "
Oct 30 23:23:58.009535 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:58.009505    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d23c612f-7a03-4b86-af74-6315ddf4657a-maas-api-openshift-ca" pod "d23c612f-7a03-4b86-af74-6315ddf4657a" (UID: "d23c612f-7a03-4b86-af74-6315ddf4657a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:23:58.011745 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:58.011709    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23c612f-7a03-4b86-af74-6315ddf4657a-kube-api-access-k6r5v" pod "d23c612f-7a03-4b86-af74-6315ddf4657a" (UID: "d23c612f-7a03-4b86-af74-6315ddf4657a"). InnerVolumeSpecName "kube-api-access-k6r5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:23:58.014729 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:58.014696    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d23c612f-7a03-4b86-af74-6315ddf4657a-maas-api-xks-ca" pod "d23c612f-7a03-4b86-af74-6315ddf4657a" (UID: "d23c612f-7a03-4b86-af74-6315ddf4657a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:23:58.078999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:58.078966    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8nrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:23:58.079230 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:58.079201    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp" podUID="84900961-7d7e-4bf0-9c40-6f3d0d8102a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 23:23:58.110440 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:58.110417    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d23c612f-7a03-4b86-af74-6315ddf4657a-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:58.110440 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:58.110440    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6r5v\" (UniqueName: \"kubernetes.io/projected/d23c612f-7a03-4b86-af74-6315ddf4657a-kube-api-access-k6r5v\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:58.110838 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:58.110450    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d23c612f-7a03-4b86-af74-6315ddf4657a-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:23:58.723963 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:58.723887    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4"
Oct 30 23:23:58.724092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:58.723883    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4" event={"ID":"d23c612f-7a03-4b86-af74-6315ddf4657a","Type":"ContainerDied","Data":"b8a94d3303ca394046437e135a2c523dd22736f7c3844c6b0ca29685817942f1"}
Oct 30 23:23:58.724092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:58.724011    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8a94d3303ca394046437e135a2c523dd22736f7c3844c6b0ca29685817942f1"
Oct 30 23:23:59.076621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:59.070921    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8nrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:23:59.076621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:23:59.070989    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp" podUID="84900961-7d7e-4bf0-9c40-6f3d0d8102a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 23:24:00.076622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:00.065940    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8nrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:24:00.076622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:00.066022    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp" podUID="84900961-7d7e-4bf0-9c40-6f3d0d8102a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 23:24:01.073623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.073008    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8nrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:24:01.073623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.073071    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp" podUID="84900961-7d7e-4bf0-9c40-6f3d0d8102a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 23:24:01.822425 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.822381    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk"]
Oct 30 23:24:01.823363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.823337    5078 cpu_manager.go:414] "containerMap: removing container" podUID="74c1fe84-7d11-49a3-b714-198724453ebc" containerName="revoke-keys"
Oct 30 23:24:01.823363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.823365    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74c1fe84-7d11-49a3-b714-198724453ebc" containerName="revoke-keys"
Oct 30 23:24:01.823504 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.823447    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d23c612f-7a03-4b86-af74-6315ddf4657a" containerName="revoke-keys"
Oct 30 23:24:01.823504 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.823455    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d23c612f-7a03-4b86-af74-6315ddf4657a" containerName="revoke-keys"
Oct 30 23:24:01.823504 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.823480    5078 cpu_manager.go:414] "containerMap: removing container" podUID="2278116c-cd84-4312-999a-832dae090303" containerName="revoke-keys"
Oct 30 23:24:01.823504 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.823487    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2278116c-cd84-4312-999a-832dae090303" containerName="revoke-keys"
Oct 30 23:24:01.823736 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.823707    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d23c612f-7a03-4b86-af74-6315ddf4657a" containerName="revoke-keys"
Oct 30 23:24:01.823736 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.823722    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="2278116c-cd84-4312-999a-832dae090303" containerName="revoke-keys"
Oct 30 23:24:01.823820 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.823739    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="74c1fe84-7d11-49a3-b714-198724453ebc" containerName="revoke-keys"
Oct 30 23:24:01.828376 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.828345    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk"
Oct 30 23:24:01.840870 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.840846    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999c0695-1f53-4b34-8331-4f569a986957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5ab077b-76nzk\" (UID: \"999c0695-1f53-4b34-8331-4f569a986957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk"
Oct 30 23:24:01.840973 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.840876    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q2gl\" (UniqueName: \"kubernetes.io/projected/999c0695-1f53-4b34-8331-4f569a986957-kube-api-access-8q2gl\") pod \"maas-api-revoke-sub-b5ab077b-76nzk\" (UID: \"999c0695-1f53-4b34-8331-4f569a986957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk"
Oct 30 23:24:01.841049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.840977    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999c0695-1f53-4b34-8331-4f569a986957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5ab077b-76nzk\" (UID: \"999c0695-1f53-4b34-8331-4f569a986957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk"
Oct 30 23:24:01.880982 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.880956    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk"]
Oct 30 23:24:01.926860 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.926822    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:24:01.926860 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.926828    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:24:01.929749 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.929729    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:24:01.929749 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.929742    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:24:01.941545 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.941524    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999c0695-1f53-4b34-8331-4f569a986957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5ab077b-76nzk\" (UID: \"999c0695-1f53-4b34-8331-4f569a986957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk"
Oct 30 23:24:01.941698 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.941551    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q2gl\" (UniqueName: \"kubernetes.io/projected/999c0695-1f53-4b34-8331-4f569a986957-kube-api-access-8q2gl\") pod \"maas-api-revoke-sub-b5ab077b-76nzk\" (UID: \"999c0695-1f53-4b34-8331-4f569a986957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk"
Oct 30 23:24:01.941698 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.941616    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999c0695-1f53-4b34-8331-4f569a986957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5ab077b-76nzk\" (UID: \"999c0695-1f53-4b34-8331-4f569a986957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk"
Oct 30 23:24:01.942171 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.942149    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999c0695-1f53-4b34-8331-4f569a986957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5ab077b-76nzk\" (UID: \"999c0695-1f53-4b34-8331-4f569a986957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk"
Oct 30 23:24:01.944069 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.944047    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999c0695-1f53-4b34-8331-4f569a986957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5ab077b-76nzk\" (UID: \"999c0695-1f53-4b34-8331-4f569a986957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk"
Oct 30 23:24:01.998616 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:01.998578    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q2gl\" (UniqueName: \"kubernetes.io/projected/999c0695-1f53-4b34-8331-4f569a986957-kube-api-access-8q2gl\") pod \"maas-api-revoke-sub-b5ab077b-76nzk\" (UID: \"999c0695-1f53-4b34-8331-4f569a986957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk"
Oct 30 23:24:02.066451 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:02.066423    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8nrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:24:02.066628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:02.066480    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp" podUID="84900961-7d7e-4bf0-9c40-6f3d0d8102a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 23:24:02.142352 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:02.142265    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk"
Oct 30 23:24:02.516822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:02.516797    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk"]
Oct 30 23:24:02.519699 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:24:02.519670    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999c0695_1f53_4b34_8331_4f569a986957.slice/crio-1c4e0d708ebc629665dc15a48371bf251f1db34230937b7b5726a13365064727 WatchSource:0}: Error finding container 1c4e0d708ebc629665dc15a48371bf251f1db34230937b7b5726a13365064727: Status 404 returned error can't find the container with id 1c4e0d708ebc629665dc15a48371bf251f1db34230937b7b5726a13365064727
Oct 30 23:24:02.748029 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:02.747993    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk" event={"ID":"999c0695-1f53-4b34-8331-4f569a986957","Type":"ContainerStarted","Data":"1c4e0d708ebc629665dc15a48371bf251f1db34230937b7b5726a13365064727"}
Oct 30 23:24:03.028035 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:03.027951    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:24:03.040810 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:03.040786    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:24:03.066637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:03.066577    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-8nrmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Oct 30 23:24:03.066762 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:03.066662    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp" podUID="84900961-7d7e-4bf0-9c40-6f3d0d8102a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Oct 30 23:24:03.756364 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:03.756284    5078 generic.go:356] "Generic (PLEG): container finished" podID="999c0695-1f53-4b34-8331-4f569a986957" containerID="8d88a4b586da6ad0702e94a33d56a089cf11728c6ae9aebfcd3b4f559836c552" exitCode=0
Oct 30 23:24:03.756517 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:03.756368    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk" event={"ID":"999c0695-1f53-4b34-8331-4f569a986957","Type":"ContainerDied","Data":"8d88a4b586da6ad0702e94a33d56a089cf11728c6ae9aebfcd3b4f559836c552"}
Oct 30 23:24:03.767994 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:03.767959    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:24:04.141088 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.141054    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:24:04.141504 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.141464    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:24:04.142524 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.142496    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:24:04.665524 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.665494    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9"]
Oct 30 23:24:04.669079 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.669060    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9"
Oct 30 23:24:04.726003 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.725975    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9"]
Oct 30 23:24:04.773788 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.773760    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5q6k\" (UniqueName: \"kubernetes.io/projected/c1905425-0b5c-4ed9-85f0-b674aaafe146-kube-api-access-w5q6k\") pod \"maas-api-revoke-sub-260604bf-sxpt9\" (UID: \"c1905425-0b5c-4ed9-85f0-b674aaafe146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9"
Oct 30 23:24:04.773936 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.773828    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1905425-0b5c-4ed9-85f0-b674aaafe146-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-260604bf-sxpt9\" (UID: \"c1905425-0b5c-4ed9-85f0-b674aaafe146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9"
Oct 30 23:24:04.773936 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.773906    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1905425-0b5c-4ed9-85f0-b674aaafe146-maas-api-xks-ca\") pod \"maas-api-revoke-sub-260604bf-sxpt9\" (UID: \"c1905425-0b5c-4ed9-85f0-b674aaafe146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9"
Oct 30 23:24:04.874564 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.874530    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1905425-0b5c-4ed9-85f0-b674aaafe146-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-260604bf-sxpt9\" (UID: \"c1905425-0b5c-4ed9-85f0-b674aaafe146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9"
Oct 30 23:24:04.874737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.874631    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1905425-0b5c-4ed9-85f0-b674aaafe146-maas-api-xks-ca\") pod \"maas-api-revoke-sub-260604bf-sxpt9\" (UID: \"c1905425-0b5c-4ed9-85f0-b674aaafe146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9"
Oct 30 23:24:04.874857 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.874838    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5q6k\" (UniqueName: \"kubernetes.io/projected/c1905425-0b5c-4ed9-85f0-b674aaafe146-kube-api-access-w5q6k\") pod \"maas-api-revoke-sub-260604bf-sxpt9\" (UID: \"c1905425-0b5c-4ed9-85f0-b674aaafe146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9"
Oct 30 23:24:04.875269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.875239    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1905425-0b5c-4ed9-85f0-b674aaafe146-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-260604bf-sxpt9\" (UID: \"c1905425-0b5c-4ed9-85f0-b674aaafe146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9"
Oct 30 23:24:04.877060 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.877035    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1905425-0b5c-4ed9-85f0-b674aaafe146-maas-api-xks-ca\") pod \"maas-api-revoke-sub-260604bf-sxpt9\" (UID: \"c1905425-0b5c-4ed9-85f0-b674aaafe146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9"
Oct 30 23:24:04.911552 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.911531    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk"
Oct 30 23:24:04.931807 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.931746    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5q6k\" (UniqueName: \"kubernetes.io/projected/c1905425-0b5c-4ed9-85f0-b674aaafe146-kube-api-access-w5q6k\") pod \"maas-api-revoke-sub-260604bf-sxpt9\" (UID: \"c1905425-0b5c-4ed9-85f0-b674aaafe146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9"
Oct 30 23:24:04.975879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.975853    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/999c0695-1f53-4b34-8331-4f569a986957-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999c0695-1f53-4b34-8331-4f569a986957-maas-api-xks-ca\") pod \"999c0695-1f53-4b34-8331-4f569a986957\" (UID: \"999c0695-1f53-4b34-8331-4f569a986957\") "
Oct 30 23:24:04.975989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.975978    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999c0695-1f53-4b34-8331-4f569a986957-kube-api-access-8q2gl\" (UniqueName: \"kubernetes.io/projected/999c0695-1f53-4b34-8331-4f569a986957-kube-api-access-8q2gl\") pod \"999c0695-1f53-4b34-8331-4f569a986957\" (UID: \"999c0695-1f53-4b34-8331-4f569a986957\") "
Oct 30 23:24:04.976070 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.976055    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999c0695-1f53-4b34-8331-4f569a986957-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999c0695-1f53-4b34-8331-4f569a986957-maas-api-openshift-ca\") pod \"999c0695-1f53-4b34-8331-4f569a986957\" (UID: \"999c0695-1f53-4b34-8331-4f569a986957\") "
Oct 30 23:24:04.976405 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.976384    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999c0695-1f53-4b34-8331-4f569a986957-maas-api-openshift-ca" pod "999c0695-1f53-4b34-8331-4f569a986957" (UID: "999c0695-1f53-4b34-8331-4f569a986957"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:24:04.977951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.977925    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/999c0695-1f53-4b34-8331-4f569a986957-maas-api-xks-ca" pod "999c0695-1f53-4b34-8331-4f569a986957" (UID: "999c0695-1f53-4b34-8331-4f569a986957"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:24:04.978157 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.978134    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999c0695-1f53-4b34-8331-4f569a986957-kube-api-access-8q2gl" pod "999c0695-1f53-4b34-8331-4f569a986957" (UID: "999c0695-1f53-4b34-8331-4f569a986957"). InnerVolumeSpecName "kube-api-access-8q2gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:24:04.981280 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:04.981261    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9"
Oct 30 23:24:05.077461 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:05.077424    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q2gl\" (UniqueName: \"kubernetes.io/projected/999c0695-1f53-4b34-8331-4f569a986957-kube-api-access-8q2gl\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:24:05.077461 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:05.077451    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/999c0695-1f53-4b34-8331-4f569a986957-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:24:05.077461 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:05.077463    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/999c0695-1f53-4b34-8331-4f569a986957-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:24:05.352359 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:05.352334    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9"]
Oct 30 23:24:05.353679 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:24:05.353651    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1905425_0b5c_4ed9_85f0_b674aaafe146.slice/crio-b3daddffa5bd46e550d2f258f71938a4604e3ef756918dac58cde75e853ff41f WatchSource:0}: Error finding container b3daddffa5bd46e550d2f258f71938a4604e3ef756918dac58cde75e853ff41f: Status 404 returned error can't find the container with id b3daddffa5bd46e550d2f258f71938a4604e3ef756918dac58cde75e853ff41f
Oct 30 23:24:05.778235 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:05.778199    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9" event={"ID":"c1905425-0b5c-4ed9-85f0-b674aaafe146","Type":"ContainerStarted","Data":"f426bfe307cf03a9876f4330514e4ab83354ed8bf962c187eeb6e7dfa2deb14c"}
Oct 30 23:24:05.778362 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:05.778256    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9" event={"ID":"c1905425-0b5c-4ed9-85f0-b674aaafe146","Type":"ContainerStarted","Data":"b3daddffa5bd46e550d2f258f71938a4604e3ef756918dac58cde75e853ff41f"}
Oct 30 23:24:05.779555 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:05.779537    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk"
Oct 30 23:24:05.779703 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:05.779586    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk" event={"ID":"999c0695-1f53-4b34-8331-4f569a986957","Type":"ContainerDied","Data":"1c4e0d708ebc629665dc15a48371bf251f1db34230937b7b5726a13365064727"}
Oct 30 23:24:05.779703 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:05.779647    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c4e0d708ebc629665dc15a48371bf251f1db34230937b7b5726a13365064727"
Oct 30 23:24:05.894342 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:05.894295    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9" podStartSLOduration=1.8942812980000001 podStartE2EDuration="1.894281298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:24:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:05.89222873 +0000 UTC m=+1031.780135083" watchObservedRunningTime="2026-10-30 23:24:05.894281298 +0000 UTC m=+1031.782187682"
Oct 30 23:24:06.786563 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:06.786530    5078 generic.go:356] "Generic (PLEG): container finished" podID="c1905425-0b5c-4ed9-85f0-b674aaafe146" containerID="f426bfe307cf03a9876f4330514e4ab83354ed8bf962c187eeb6e7dfa2deb14c" exitCode=0
Oct 30 23:24:06.787007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:06.786635    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9" event={"ID":"c1905425-0b5c-4ed9-85f0-b674aaafe146","Type":"ContainerDied","Data":"f426bfe307cf03a9876f4330514e4ab83354ed8bf962c187eeb6e7dfa2deb14c"}
Oct 30 23:24:07.953113 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:07.953088    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9"
Oct 30 23:24:08.028067 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.028040    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1905425-0b5c-4ed9-85f0-b674aaafe146-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1905425-0b5c-4ed9-85f0-b674aaafe146-maas-api-openshift-ca\") pod \"c1905425-0b5c-4ed9-85f0-b674aaafe146\" (UID: \"c1905425-0b5c-4ed9-85f0-b674aaafe146\") "
Oct 30 23:24:08.028187 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.028107    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1905425-0b5c-4ed9-85f0-b674aaafe146-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1905425-0b5c-4ed9-85f0-b674aaafe146-maas-api-xks-ca\") pod \"c1905425-0b5c-4ed9-85f0-b674aaafe146\" (UID: \"c1905425-0b5c-4ed9-85f0-b674aaafe146\") "
Oct 30 23:24:08.028187 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.028146    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1905425-0b5c-4ed9-85f0-b674aaafe146-kube-api-access-w5q6k\" (UniqueName: \"kubernetes.io/projected/c1905425-0b5c-4ed9-85f0-b674aaafe146-kube-api-access-w5q6k\") pod \"c1905425-0b5c-4ed9-85f0-b674aaafe146\" (UID: \"c1905425-0b5c-4ed9-85f0-b674aaafe146\") "
Oct 30 23:24:08.028380 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.028355    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1905425-0b5c-4ed9-85f0-b674aaafe146-maas-api-openshift-ca" pod "c1905425-0b5c-4ed9-85f0-b674aaafe146" (UID: "c1905425-0b5c-4ed9-85f0-b674aaafe146"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:24:08.028520 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.028498    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1905425-0b5c-4ed9-85f0-b674aaafe146-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:24:08.030336 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.030316    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1905425-0b5c-4ed9-85f0-b674aaafe146-maas-api-xks-ca" pod "c1905425-0b5c-4ed9-85f0-b674aaafe146" (UID: "c1905425-0b5c-4ed9-85f0-b674aaafe146"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:24:08.030414 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.030377    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1905425-0b5c-4ed9-85f0-b674aaafe146-kube-api-access-w5q6k" pod "c1905425-0b5c-4ed9-85f0-b674aaafe146" (UID: "c1905425-0b5c-4ed9-85f0-b674aaafe146"). InnerVolumeSpecName "kube-api-access-w5q6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:24:08.129658 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.129548    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1905425-0b5c-4ed9-85f0-b674aaafe146-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:24:08.129658 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.129577    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5q6k\" (UniqueName: \"kubernetes.io/projected/c1905425-0b5c-4ed9-85f0-b674aaafe146-kube-api-access-w5q6k\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:24:08.556731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.556694    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w"]
Oct 30 23:24:08.557163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.557150    5078 cpu_manager.go:414] "containerMap: removing container" podUID="999c0695-1f53-4b34-8331-4f569a986957" containerName="revoke-keys"
Oct 30 23:24:08.557207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.557165    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999c0695-1f53-4b34-8331-4f569a986957" containerName="revoke-keys"
Oct 30 23:24:08.557207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.557196    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c1905425-0b5c-4ed9-85f0-b674aaafe146" containerName="revoke-keys"
Oct 30 23:24:08.557207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.557201    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1905425-0b5c-4ed9-85f0-b674aaafe146" containerName="revoke-keys"
Oct 30 23:24:08.557298 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.557263    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1905425-0b5c-4ed9-85f0-b674aaafe146" containerName="revoke-keys"
Oct 30 23:24:08.557298 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.557273    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="999c0695-1f53-4b34-8331-4f569a986957" containerName="revoke-keys"
Oct 30 23:24:08.560543 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.560525    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w"
Oct 30 23:24:08.617970 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.617932    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w"]
Oct 30 23:24:08.634067 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.634033    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spczr\" (UniqueName: \"kubernetes.io/projected/51e829e1-3829-40dd-93d2-ef0d55b74127-kube-api-access-spczr\") pod \"maas-api-revoke-sub-dbb516cb-t584w\" (UID: \"51e829e1-3829-40dd-93d2-ef0d55b74127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w"
Oct 30 23:24:08.634243 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.634086    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e829e1-3829-40dd-93d2-ef0d55b74127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbb516cb-t584w\" (UID: \"51e829e1-3829-40dd-93d2-ef0d55b74127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w"
Oct 30 23:24:08.634243 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.634226    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e829e1-3829-40dd-93d2-ef0d55b74127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbb516cb-t584w\" (UID: \"51e829e1-3829-40dd-93d2-ef0d55b74127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w"
Oct 30 23:24:08.735486 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.735448    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spczr\" (UniqueName: \"kubernetes.io/projected/51e829e1-3829-40dd-93d2-ef0d55b74127-kube-api-access-spczr\") pod \"maas-api-revoke-sub-dbb516cb-t584w\" (UID: \"51e829e1-3829-40dd-93d2-ef0d55b74127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w"
Oct 30 23:24:08.735697 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.735497    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e829e1-3829-40dd-93d2-ef0d55b74127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbb516cb-t584w\" (UID: \"51e829e1-3829-40dd-93d2-ef0d55b74127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w"
Oct 30 23:24:08.735697 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.735556    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e829e1-3829-40dd-93d2-ef0d55b74127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbb516cb-t584w\" (UID: \"51e829e1-3829-40dd-93d2-ef0d55b74127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w"
Oct 30 23:24:08.736349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.736326    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e829e1-3829-40dd-93d2-ef0d55b74127-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbb516cb-t584w\" (UID: \"51e829e1-3829-40dd-93d2-ef0d55b74127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w"
Oct 30 23:24:08.738191 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.738173    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e829e1-3829-40dd-93d2-ef0d55b74127-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbb516cb-t584w\" (UID: \"51e829e1-3829-40dd-93d2-ef0d55b74127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w"
Oct 30 23:24:08.792975 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.792944    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spczr\" (UniqueName: \"kubernetes.io/projected/51e829e1-3829-40dd-93d2-ef0d55b74127-kube-api-access-spczr\") pod \"maas-api-revoke-sub-dbb516cb-t584w\" (UID: \"51e829e1-3829-40dd-93d2-ef0d55b74127\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w"
Oct 30 23:24:08.797948 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.797923    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9"
Oct 30 23:24:08.798090 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.797925    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9" event={"ID":"c1905425-0b5c-4ed9-85f0-b674aaafe146","Type":"ContainerDied","Data":"b3daddffa5bd46e550d2f258f71938a4604e3ef756918dac58cde75e853ff41f"}
Oct 30 23:24:08.798090 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.798057    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3daddffa5bd46e550d2f258f71938a4604e3ef756918dac58cde75e853ff41f"
Oct 30 23:24:08.874928 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:08.874849    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w"
Oct 30 23:24:09.248684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:09.248655    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w"]
Oct 30 23:24:09.249096 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:24:09.248986    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e829e1_3829_40dd_93d2_ef0d55b74127.slice/crio-f5b422d8f9dc0321510ece38f57d57402880260db17a1da50e3fbbe2fd6e86f1 WatchSource:0}: Error finding container f5b422d8f9dc0321510ece38f57d57402880260db17a1da50e3fbbe2fd6e86f1: Status 404 returned error can't find the container with id f5b422d8f9dc0321510ece38f57d57402880260db17a1da50e3fbbe2fd6e86f1
Oct 30 23:24:09.805247 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:09.805212    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w" event={"ID":"51e829e1-3829-40dd-93d2-ef0d55b74127","Type":"ContainerStarted","Data":"bd8b8673aa51417e5bad81a40878abd6522b9f499701a7bd7f28fbde4c45942d"}
Oct 30 23:24:09.805423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:09.805266    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w" event={"ID":"51e829e1-3829-40dd-93d2-ef0d55b74127","Type":"ContainerStarted","Data":"f5b422d8f9dc0321510ece38f57d57402880260db17a1da50e3fbbe2fd6e86f1"}
Oct 30 23:24:09.912501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:09.912422    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"]
Oct 30 23:24:09.916060 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:09.916044    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:09.924362 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:09.924325    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w" podStartSLOduration=1.924313235 podStartE2EDuration="1.924313235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:24:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:09.921050634 +0000 UTC m=+1035.808956987" watchObservedRunningTime="2026-10-30 23:24:09.924313235 +0000 UTC m=+1035.812219588"
Oct 30 23:24:10.010545 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.010521    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"]
Oct 30 23:24:10.051473 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.051441    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.051662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.051480    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.051662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.051631    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.051787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.051662    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a363bd-8ee8-4644-b685-e24ddfa39788-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.051787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.051683    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.051893 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.051804    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.051893 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.051864    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.051996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.051906    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttmwf\" (UniqueName: \"kubernetes.io/projected/a0a363bd-8ee8-4644-b685-e24ddfa39788-kube-api-access-ttmwf\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.052054 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.052007    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.153649 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.153623    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.153787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.153664    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a363bd-8ee8-4644-b685-e24ddfa39788-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.153787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.153683    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.153787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.153708    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.153787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.153740    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.153787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.153778    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttmwf\" (UniqueName: \"kubernetes.io/projected/a0a363bd-8ee8-4644-b685-e24ddfa39788-kube-api-access-ttmwf\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.154057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.153838    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.154057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.153868    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.154057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.153907    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.154057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.153985    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.154259 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.154086    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.154319 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.154256    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.154319 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.154290    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.154396 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.154317    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a363bd-8ee8-4644-b685-e24ddfa39788-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.156023 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.156005    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.156398 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.156382    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.210731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.210674    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.210825 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.210729    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttmwf\" (UniqueName: \"kubernetes.io/projected/a0a363bd-8ee8-4644-b685-e24ddfa39788-kube-api-access-ttmwf\") pod \"maas-default-gateway-openshift-default-5df48fcc99-s9q2m\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.229401 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.229383    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:10.650727 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.650703    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"]
Oct 30 23:24:10.651529 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:24:10.651502    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a363bd_8ee8_4644_b685_e24ddfa39788.slice/crio-4145bd7e800391e1eb862a2ac601da2917525550c3c51404c86e63d1dd0dc786 WatchSource:0}: Error finding container 4145bd7e800391e1eb862a2ac601da2917525550c3c51404c86e63d1dd0dc786: Status 404 returned error can't find the container with id 4145bd7e800391e1eb862a2ac601da2917525550c3c51404c86e63d1dd0dc786
Oct 30 23:24:10.653895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.653865    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:24:10.653977 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.653926    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:24:10.653977 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.653952    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:24:10.812557 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.812500    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m" event={"ID":"a0a363bd-8ee8-4644-b685-e24ddfa39788","Type":"ContainerStarted","Data":"930ef0b0be8f46366e681b7d050945d226e432828f3b401aab8d790b244dc0c8"}
Oct 30 23:24:10.812758 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.812568    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m" event={"ID":"a0a363bd-8ee8-4644-b685-e24ddfa39788","Type":"ContainerStarted","Data":"4145bd7e800391e1eb862a2ac601da2917525550c3c51404c86e63d1dd0dc786"}
Oct 30 23:24:10.814240 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.814208    5078 generic.go:356] "Generic (PLEG): container finished" podID="51e829e1-3829-40dd-93d2-ef0d55b74127" containerID="bd8b8673aa51417e5bad81a40878abd6522b9f499701a7bd7f28fbde4c45942d" exitCode=0
Oct 30 23:24:10.814348 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.814281    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w" event={"ID":"51e829e1-3829-40dd-93d2-ef0d55b74127","Type":"ContainerDied","Data":"bd8b8673aa51417e5bad81a40878abd6522b9f499701a7bd7f28fbde4c45942d"}
Oct 30 23:24:10.934077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:10.933985    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m" podStartSLOduration=1.933970172 podStartE2EDuration="1.933970172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:24:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:10.932165695 +0000 UTC m=+1036.820072048" watchObservedRunningTime="2026-10-30 23:24:10.933970172 +0000 UTC m=+1036.821876526"
Oct 30 23:24:11.230426 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:11.230343    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:11.232162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:11.232133    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-s9q2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 23:24:11.232304 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:11.232189    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m" podUID="a0a363bd-8ee8-4644-b685-e24ddfa39788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 23:24:11.442065 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:11.442034    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk"]
Oct 30 23:24:11.445732 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:11.445715    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk"
Oct 30 23:24:11.502657 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:11.502573    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk"]
Oct 30 23:24:11.586294 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:11.586260    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkxcv\" (UniqueName: \"kubernetes.io/projected/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-kube-api-access-zkxcv\") pod \"maas-api-revoke-sub-8bff4dcb-sdlqk\" (UID: \"411a9ab4-493f-4e0f-9e11-b2fb0db1618d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk"
Oct 30 23:24:11.586466 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:11.586303    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bff4dcb-sdlqk\" (UID: \"411a9ab4-493f-4e0f-9e11-b2fb0db1618d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk"
Oct 30 23:24:11.586466 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:11.586322    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bff4dcb-sdlqk\" (UID: \"411a9ab4-493f-4e0f-9e11-b2fb0db1618d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk"
Oct 30 23:24:11.687464 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:11.687426    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkxcv\" (UniqueName: \"kubernetes.io/projected/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-kube-api-access-zkxcv\") pod \"maas-api-revoke-sub-8bff4dcb-sdlqk\" (UID: \"411a9ab4-493f-4e0f-9e11-b2fb0db1618d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk"
Oct 30 23:24:11.687994 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:11.687484    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bff4dcb-sdlqk\" (UID: \"411a9ab4-493f-4e0f-9e11-b2fb0db1618d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk"
Oct 30 23:24:11.687994 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:11.687511    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bff4dcb-sdlqk\" (UID: \"411a9ab4-493f-4e0f-9e11-b2fb0db1618d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk"
Oct 30 23:24:11.688251 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:11.688227    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bff4dcb-sdlqk\" (UID: \"411a9ab4-493f-4e0f-9e11-b2fb0db1618d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk"
Oct 30 23:24:11.690183 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:11.690159    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bff4dcb-sdlqk\" (UID: \"411a9ab4-493f-4e0f-9e11-b2fb0db1618d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk"
Oct 30 23:24:11.744963 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:11.744934    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkxcv\" (UniqueName: \"kubernetes.io/projected/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-kube-api-access-zkxcv\") pod \"maas-api-revoke-sub-8bff4dcb-sdlqk\" (UID: \"411a9ab4-493f-4e0f-9e11-b2fb0db1618d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk"
Oct 30 23:24:11.781892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:11.781792    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk"
Oct 30 23:24:12.023565 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.023542    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w"
Oct 30 23:24:12.093181 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.093154    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e829e1-3829-40dd-93d2-ef0d55b74127-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e829e1-3829-40dd-93d2-ef0d55b74127-maas-api-openshift-ca\") pod \"51e829e1-3829-40dd-93d2-ef0d55b74127\" (UID: \"51e829e1-3829-40dd-93d2-ef0d55b74127\") "
Oct 30 23:24:12.093343 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.093265    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e829e1-3829-40dd-93d2-ef0d55b74127-kube-api-access-spczr\" (UniqueName: \"kubernetes.io/projected/51e829e1-3829-40dd-93d2-ef0d55b74127-kube-api-access-spczr\") pod \"51e829e1-3829-40dd-93d2-ef0d55b74127\" (UID: \"51e829e1-3829-40dd-93d2-ef0d55b74127\") "
Oct 30 23:24:12.093343 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.093315    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51e829e1-3829-40dd-93d2-ef0d55b74127-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e829e1-3829-40dd-93d2-ef0d55b74127-maas-api-xks-ca\") pod \"51e829e1-3829-40dd-93d2-ef0d55b74127\" (UID: \"51e829e1-3829-40dd-93d2-ef0d55b74127\") "
Oct 30 23:24:12.093591 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.093570    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e829e1-3829-40dd-93d2-ef0d55b74127-maas-api-openshift-ca" pod "51e829e1-3829-40dd-93d2-ef0d55b74127" (UID: "51e829e1-3829-40dd-93d2-ef0d55b74127"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:24:12.093740 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.093726    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51e829e1-3829-40dd-93d2-ef0d55b74127-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:24:12.095427 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.095402    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e829e1-3829-40dd-93d2-ef0d55b74127-kube-api-access-spczr" pod "51e829e1-3829-40dd-93d2-ef0d55b74127" (UID: "51e829e1-3829-40dd-93d2-ef0d55b74127"). InnerVolumeSpecName "kube-api-access-spczr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:24:12.095507 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.095486    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e829e1-3829-40dd-93d2-ef0d55b74127-maas-api-xks-ca" pod "51e829e1-3829-40dd-93d2-ef0d55b74127" (UID: "51e829e1-3829-40dd-93d2-ef0d55b74127"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:24:12.228989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.173587    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk"]
Oct 30 23:24:12.228989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.195461    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spczr\" (UniqueName: \"kubernetes.io/projected/51e829e1-3829-40dd-93d2-ef0d55b74127-kube-api-access-spczr\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:24:12.228989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.195492    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51e829e1-3829-40dd-93d2-ef0d55b74127-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:24:12.235640 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.234853    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-s9q2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 23:24:12.235640 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.234925    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m" podUID="a0a363bd-8ee8-4644-b685-e24ddfa39788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 23:24:12.829466 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.829433    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk" event={"ID":"411a9ab4-493f-4e0f-9e11-b2fb0db1618d","Type":"ContainerStarted","Data":"7f7e64bf704c513437d5016a50a4ecaa194eb452f7159e14239ee309490cb858"}
Oct 30 23:24:12.829928 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.829493    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk" event={"ID":"411a9ab4-493f-4e0f-9e11-b2fb0db1618d","Type":"ContainerStarted","Data":"2da512b7b6a64671b707b3c3a12db5e9ee38d903f4cc1b6b2ee6629f8d11a0d5"}
Oct 30 23:24:12.830852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.830827    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w"
Oct 30 23:24:12.830959 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.830863    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w" event={"ID":"51e829e1-3829-40dd-93d2-ef0d55b74127","Type":"ContainerDied","Data":"f5b422d8f9dc0321510ece38f57d57402880260db17a1da50e3fbbe2fd6e86f1"}
Oct 30 23:24:12.830959 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.830904    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5b422d8f9dc0321510ece38f57d57402880260db17a1da50e3fbbe2fd6e86f1"
Oct 30 23:24:12.885487 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.885451    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"]
Oct 30 23:24:12.885914 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.885901    5078 cpu_manager.go:414] "containerMap: removing container" podUID="51e829e1-3829-40dd-93d2-ef0d55b74127" containerName="revoke-keys"
Oct 30 23:24:12.885914 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.885916    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e829e1-3829-40dd-93d2-ef0d55b74127" containerName="revoke-keys"
Oct 30 23:24:12.886018 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.886009    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e829e1-3829-40dd-93d2-ef0d55b74127" containerName="revoke-keys"
Oct 30 23:24:12.889298 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.889281    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:12.935724 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.935692    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"]
Oct 30 23:24:12.944093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.944066    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b81a7a-openshift-default-dockercfg-btsst\""
Oct 30 23:24:12.945978 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:12.945938    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk" podStartSLOduration=1.9459231099999998 podStartE2EDuration="1.94592311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:12.944200995 +0000 UTC m=+1038.832107348" watchObservedRunningTime="2026-10-30 23:24:12.94592311 +0000 UTC m=+1038.833829463"
Oct 30 23:24:13.002885 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.002852    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-workload-certs\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.003061 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.002889    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1256cc85-9025-4545-949b-c37d21d8b052-istio-token\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.003061 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.002952    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-istio-data\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.003061 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.003032    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-istio-envoy\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.003200 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.003170    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-workload-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.003232 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.003209    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pp77\" (UniqueName: \"kubernetes.io/projected/1256cc85-9025-4545-949b-c37d21d8b052-kube-api-access-2pp77\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.003266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.003239    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1256cc85-9025-4545-949b-c37d21d8b052-istio-podinfo\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.003301 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.003284    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-credential-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.003368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.003352    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1256cc85-9025-4545-949b-c37d21d8b052-istiod-ca-cert\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.104891 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.104816    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-workload-certs\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.104891 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.104848    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1256cc85-9025-4545-949b-c37d21d8b052-istio-token\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.104891 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.104881    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-istio-data\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.105127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.104991    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-istio-envoy\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.105127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.105041    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-workload-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.105127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.105073    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pp77\" (UniqueName: \"kubernetes.io/projected/1256cc85-9025-4545-949b-c37d21d8b052-kube-api-access-2pp77\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.105127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.105102    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1256cc85-9025-4545-949b-c37d21d8b052-istio-podinfo\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.105325 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.105153    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-credential-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.105325 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.105180    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-istio-data\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.105325 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.105248    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1256cc85-9025-4545-949b-c37d21d8b052-istiod-ca-cert\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.105325 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.105302    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-workload-certs\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.105526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.105459    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-credential-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.105587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.105532    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-workload-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.105902 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.105876    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1256cc85-9025-4545-949b-c37d21d8b052-istiod-ca-cert\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.107285 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.107264    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-istio-envoy\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.107560 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.107543    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1256cc85-9025-4545-949b-c37d21d8b052-istio-podinfo\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.184288 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.184252    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pp77\" (UniqueName: \"kubernetes.io/projected/1256cc85-9025-4545-949b-c37d21d8b052-kube-api-access-2pp77\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.185564 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.185527    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1256cc85-9025-4545-949b-c37d21d8b052-istio-token\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.202601 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.202577    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:13.230802 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.230770    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-s9q2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 23:24:13.230918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.230835    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m" podUID="a0a363bd-8ee8-4644-b685-e24ddfa39788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 23:24:13.652992 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.652967    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"]
Oct 30 23:24:13.654249 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:24:13.654219    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1256cc85_9025_4545_949b_c37d21d8b052.slice/crio-90f13a5d2037a0cc234c58ac2675336825397b1629c448e48f1b6cad26914f74 WatchSource:0}: Error finding container 90f13a5d2037a0cc234c58ac2675336825397b1629c448e48f1b6cad26914f74: Status 404 returned error can't find the container with id 90f13a5d2037a0cc234c58ac2675336825397b1629c448e48f1b6cad26914f74
Oct 30 23:24:13.671633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.656831    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:24:13.671633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.656915    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:24:13.671633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.656959    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:24:13.840910 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.840877    5078 generic.go:356] "Generic (PLEG): container finished" podID="411a9ab4-493f-4e0f-9e11-b2fb0db1618d" containerID="7f7e64bf704c513437d5016a50a4ecaa194eb452f7159e14239ee309490cb858" exitCode=0
Oct 30 23:24:13.841377 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.841002    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk" event={"ID":"411a9ab4-493f-4e0f-9e11-b2fb0db1618d","Type":"ContainerDied","Data":"7f7e64bf704c513437d5016a50a4ecaa194eb452f7159e14239ee309490cb858"}
Oct 30 23:24:13.842891 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.842867    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs" event={"ID":"1256cc85-9025-4545-949b-c37d21d8b052","Type":"ContainerStarted","Data":"f00b06c2aa8e5dd417e596760b95b3a6cac8e63a0e096fbe2250a07d868cf18b"}
Oct 30 23:24:13.843018 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:13.842911    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs" event={"ID":"1256cc85-9025-4545-949b-c37d21d8b052","Type":"ContainerStarted","Data":"90f13a5d2037a0cc234c58ac2675336825397b1629c448e48f1b6cad26914f74"}
Oct 30 23:24:14.075878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:14.075811    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs" podStartSLOduration=2.075791185 podStartE2EDuration="2.075791185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:14.071067416 +0000 UTC m=+1039.958973793" watchObservedRunningTime="2026-10-30 23:24:14.075791185 +0000 UTC m=+1039.963697541"
Oct 30 23:24:14.203412 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:14.203378    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:14.204658 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:14.204628    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:24:14.204789 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:14.204688    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs" podUID="1256cc85-9025-4545-949b-c37d21d8b052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 23:24:14.230324 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:14.230295    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-s9q2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 23:24:14.230432 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:14.230344    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m" podUID="a0a363bd-8ee8-4644-b685-e24ddfa39788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 23:24:15.117365 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:15.117297    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk"
Oct 30 23:24:15.207015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:15.206968    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:24:15.207212 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:15.207029    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs" podUID="1256cc85-9025-4545-949b-c37d21d8b052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 23:24:15.226255 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:15.226231    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-kube-api-access-zkxcv\" (UniqueName: \"kubernetes.io/projected/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-kube-api-access-zkxcv\") pod \"411a9ab4-493f-4e0f-9e11-b2fb0db1618d\" (UID: \"411a9ab4-493f-4e0f-9e11-b2fb0db1618d\") "
Oct 30 23:24:15.226435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:15.226382    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-maas-api-xks-ca\") pod \"411a9ab4-493f-4e0f-9e11-b2fb0db1618d\" (UID: \"411a9ab4-493f-4e0f-9e11-b2fb0db1618d\") "
Oct 30 23:24:15.226435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:15.226413    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-maas-api-openshift-ca\") pod \"411a9ab4-493f-4e0f-9e11-b2fb0db1618d\" (UID: \"411a9ab4-493f-4e0f-9e11-b2fb0db1618d\") "
Oct 30 23:24:15.226961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:15.226937    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-maas-api-openshift-ca" pod "411a9ab4-493f-4e0f-9e11-b2fb0db1618d" (UID: "411a9ab4-493f-4e0f-9e11-b2fb0db1618d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:24:15.228449 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:15.228429    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-kube-api-access-zkxcv" pod "411a9ab4-493f-4e0f-9e11-b2fb0db1618d" (UID: "411a9ab4-493f-4e0f-9e11-b2fb0db1618d"). InnerVolumeSpecName "kube-api-access-zkxcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:24:15.228523 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:15.228446    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-maas-api-xks-ca" pod "411a9ab4-493f-4e0f-9e11-b2fb0db1618d" (UID: "411a9ab4-493f-4e0f-9e11-b2fb0db1618d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:24:15.230813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:15.230792    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-s9q2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 23:24:15.230876 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:15.230840    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m" podUID="a0a363bd-8ee8-4644-b685-e24ddfa39788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 23:24:15.327628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:15.327577    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:24:15.327628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:15.327624    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:24:15.327628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:15.327635    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkxcv\" (UniqueName: \"kubernetes.io/projected/411a9ab4-493f-4e0f-9e11-b2fb0db1618d-kube-api-access-zkxcv\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:24:15.898625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:15.878296    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk" event={"ID":"411a9ab4-493f-4e0f-9e11-b2fb0db1618d","Type":"ContainerDied","Data":"2da512b7b6a64671b707b3c3a12db5e9ee38d903f4cc1b6b2ee6629f8d11a0d5"}
Oct 30 23:24:15.898909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:15.898887    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2da512b7b6a64671b707b3c3a12db5e9ee38d903f4cc1b6b2ee6629f8d11a0d5"
Oct 30 23:24:15.899022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:15.879267    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk"
Oct 30 23:24:16.207132 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:16.207035    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:24:16.207132 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:16.207102    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs" podUID="1256cc85-9025-4545-949b-c37d21d8b052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 23:24:16.230660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:16.230629    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-s9q2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 23:24:16.230813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:16.230689    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m" podUID="a0a363bd-8ee8-4644-b685-e24ddfa39788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 23:24:17.203574 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:17.203543    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:24:17.203780 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:17.203615    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs" podUID="1256cc85-9025-4545-949b-c37d21d8b052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 23:24:17.230279 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:17.230239    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-s9q2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 23:24:17.230669 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:17.230294    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m" podUID="a0a363bd-8ee8-4644-b685-e24ddfa39788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 23:24:17.723818 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:24:17.723772    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:24:18.205014 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:18.204979    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:24:18.205327 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:18.205291    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs" podUID="1256cc85-9025-4545-949b-c37d21d8b052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 23:24:18.233022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:18.232984    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-s9q2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body=
Oct 30 23:24:18.233478 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:18.233042    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m" podUID="a0a363bd-8ee8-4644-b685-e24ddfa39788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused"
Oct 30 23:24:19.208668 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:19.206149    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:24:19.208668 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:19.206225    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs" podUID="1256cc85-9025-4545-949b-c37d21d8b052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 23:24:20.086487 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:20.086457    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:20.087261 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:20.087244    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:20.088760 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:20.088718    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:24:20.203201 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:20.203161    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body=
Oct 30 23:24:20.203377 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:20.203219    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs" podUID="1256cc85-9025-4545-949b-c37d21d8b052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused"
Oct 30 23:24:21.098711 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.098676    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw"]
Oct 30 23:24:21.099380 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.099358    5078 cpu_manager.go:414] "containerMap: removing container" podUID="411a9ab4-493f-4e0f-9e11-b2fb0db1618d" containerName="revoke-keys"
Oct 30 23:24:21.099482 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.099381    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="411a9ab4-493f-4e0f-9e11-b2fb0db1618d" containerName="revoke-keys"
Oct 30 23:24:21.099534 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.099496    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="411a9ab4-493f-4e0f-9e11-b2fb0db1618d" containerName="revoke-keys"
Oct 30 23:24:21.102957 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.102936    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw"
Oct 30 23:24:21.151957 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.151921    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e1d2e37-4880-49ae-85ad-b24b06578beb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35aede95-8t9sw\" (UID: \"4e1d2e37-4880-49ae-85ad-b24b06578beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw"
Oct 30 23:24:21.152137 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.151981    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e1d2e37-4880-49ae-85ad-b24b06578beb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35aede95-8t9sw\" (UID: \"4e1d2e37-4880-49ae-85ad-b24b06578beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw"
Oct 30 23:24:21.152137 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.152110    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrw8g\" (UniqueName: \"kubernetes.io/projected/4e1d2e37-4880-49ae-85ad-b24b06578beb-kube-api-access-lrw8g\") pod \"maas-api-revoke-sub-35aede95-8t9sw\" (UID: \"4e1d2e37-4880-49ae-85ad-b24b06578beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw"
Oct 30 23:24:21.155747 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.155723    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:24:21.155939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.155868    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:24:21.156069 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.156051    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:24:21.205467 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.205436    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:24:21.208695 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.208663    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw"]
Oct 30 23:24:21.253780 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.253753    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e1d2e37-4880-49ae-85ad-b24b06578beb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35aede95-8t9sw\" (UID: \"4e1d2e37-4880-49ae-85ad-b24b06578beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw"
Oct 30 23:24:21.253928 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.253805    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e1d2e37-4880-49ae-85ad-b24b06578beb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35aede95-8t9sw\" (UID: \"4e1d2e37-4880-49ae-85ad-b24b06578beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw"
Oct 30 23:24:21.253928 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.253835    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrw8g\" (UniqueName: \"kubernetes.io/projected/4e1d2e37-4880-49ae-85ad-b24b06578beb-kube-api-access-lrw8g\") pod \"maas-api-revoke-sub-35aede95-8t9sw\" (UID: \"4e1d2e37-4880-49ae-85ad-b24b06578beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw"
Oct 30 23:24:21.254379 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.254360    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e1d2e37-4880-49ae-85ad-b24b06578beb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35aede95-8t9sw\" (UID: \"4e1d2e37-4880-49ae-85ad-b24b06578beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw"
Oct 30 23:24:21.256387 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.256357    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e1d2e37-4880-49ae-85ad-b24b06578beb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35aede95-8t9sw\" (UID: \"4e1d2e37-4880-49ae-85ad-b24b06578beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw"
Oct 30 23:24:21.310748 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.310707    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrw8g\" (UniqueName: \"kubernetes.io/projected/4e1d2e37-4880-49ae-85ad-b24b06578beb-kube-api-access-lrw8g\") pod \"maas-api-revoke-sub-35aede95-8t9sw\" (UID: \"4e1d2e37-4880-49ae-85ad-b24b06578beb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw"
Oct 30 23:24:21.417082 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.416981    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw"
Oct 30 23:24:21.809514 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.809488    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw"]
Oct 30 23:24:21.811106 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:24:21.811065    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e1d2e37_4880_49ae_85ad_b24b06578beb.slice/crio-fa95b30cd7c1b91b1811fece2300db9105399198ffda43a2e72e5465d85a0bb2 WatchSource:0}: Error finding container fa95b30cd7c1b91b1811fece2300db9105399198ffda43a2e72e5465d85a0bb2: Status 404 returned error can't find the container with id fa95b30cd7c1b91b1811fece2300db9105399198ffda43a2e72e5465d85a0bb2
Oct 30 23:24:21.844519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.844492    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:21.844977 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.844957    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:21.846446 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.846425    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:24:21.962066 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:21.962008    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw" event={"ID":"4e1d2e37-4880-49ae-85ad-b24b06578beb","Type":"ContainerStarted","Data":"fa95b30cd7c1b91b1811fece2300db9105399198ffda43a2e72e5465d85a0bb2"}
Oct 30 23:24:22.969900 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:22.969805    5078 generic.go:356] "Generic (PLEG): container finished" podID="4e1d2e37-4880-49ae-85ad-b24b06578beb" containerID="e502c06a4701e9004579cf410f9ee7ee6aa54d9db478c9a76882bb40ec5b4c8b" exitCode=0
Oct 30 23:24:22.969900 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:22.969887    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw" event={"ID":"4e1d2e37-4880-49ae-85ad-b24b06578beb","Type":"ContainerDied","Data":"e502c06a4701e9004579cf410f9ee7ee6aa54d9db478c9a76882bb40ec5b4c8b"}
Oct 30 23:24:24.112079 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:24.112056    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw"
Oct 30 23:24:24.191879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:24.191850    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e1d2e37-4880-49ae-85ad-b24b06578beb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e1d2e37-4880-49ae-85ad-b24b06578beb-maas-api-xks-ca\") pod \"4e1d2e37-4880-49ae-85ad-b24b06578beb\" (UID: \"4e1d2e37-4880-49ae-85ad-b24b06578beb\") "
Oct 30 23:24:24.192045 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:24.191908    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e1d2e37-4880-49ae-85ad-b24b06578beb-kube-api-access-lrw8g\" (UniqueName: \"kubernetes.io/projected/4e1d2e37-4880-49ae-85ad-b24b06578beb-kube-api-access-lrw8g\") pod \"4e1d2e37-4880-49ae-85ad-b24b06578beb\" (UID: \"4e1d2e37-4880-49ae-85ad-b24b06578beb\") "
Oct 30 23:24:24.192112 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:24.192094    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e1d2e37-4880-49ae-85ad-b24b06578beb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e1d2e37-4880-49ae-85ad-b24b06578beb-maas-api-openshift-ca\") pod \"4e1d2e37-4880-49ae-85ad-b24b06578beb\" (UID: \"4e1d2e37-4880-49ae-85ad-b24b06578beb\") "
Oct 30 23:24:24.192421 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:24.192396    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e1d2e37-4880-49ae-85ad-b24b06578beb-maas-api-openshift-ca" pod "4e1d2e37-4880-49ae-85ad-b24b06578beb" (UID: "4e1d2e37-4880-49ae-85ad-b24b06578beb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:24:24.192553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:24.192539    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4e1d2e37-4880-49ae-85ad-b24b06578beb-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:24:24.194785 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:24.194759    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e1d2e37-4880-49ae-85ad-b24b06578beb-maas-api-xks-ca" pod "4e1d2e37-4880-49ae-85ad-b24b06578beb" (UID: "4e1d2e37-4880-49ae-85ad-b24b06578beb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:24:24.194785 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:24.194781    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1d2e37-4880-49ae-85ad-b24b06578beb-kube-api-access-lrw8g" pod "4e1d2e37-4880-49ae-85ad-b24b06578beb" (UID: "4e1d2e37-4880-49ae-85ad-b24b06578beb"). InnerVolumeSpecName "kube-api-access-lrw8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:24:24.293893 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:24.293860    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4e1d2e37-4880-49ae-85ad-b24b06578beb-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:24:24.293893 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:24.293888    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrw8g\" (UniqueName: \"kubernetes.io/projected/4e1d2e37-4880-49ae-85ad-b24b06578beb-kube-api-access-lrw8g\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:24:24.983415 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:24.983388    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw"
Oct 30 23:24:24.983570 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:24.983426    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw" event={"ID":"4e1d2e37-4880-49ae-85ad-b24b06578beb","Type":"ContainerDied","Data":"fa95b30cd7c1b91b1811fece2300db9105399198ffda43a2e72e5465d85a0bb2"}
Oct 30 23:24:24.983570 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:24.983471    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa95b30cd7c1b91b1811fece2300db9105399198ffda43a2e72e5465d85a0bb2"
Oct 30 23:24:26.029259 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.029227    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"]
Oct 30 23:24:26.029737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.029724    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4e1d2e37-4880-49ae-85ad-b24b06578beb" containerName="revoke-keys"
Oct 30 23:24:26.029809 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.029739    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e1d2e37-4880-49ae-85ad-b24b06578beb" containerName="revoke-keys"
Oct 30 23:24:26.029848 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.029811    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e1d2e37-4880-49ae-85ad-b24b06578beb" containerName="revoke-keys"
Oct 30 23:24:26.033139 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.033124    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.085354 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.085329    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-057303-openshift-default-dockercfg-gz47w\""
Oct 30 23:24:26.112872 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.112839    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-istio-envoy\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.113008 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.112890    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-istio-data\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.113075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.113012    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d861cce2-0e36-40ef-811f-e56b675031d8-istiod-ca-cert\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.113075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.113044    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lr9z\" (UniqueName: \"kubernetes.io/projected/d861cce2-0e36-40ef-811f-e56b675031d8-kube-api-access-2lr9z\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.113199 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.113180    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d861cce2-0e36-40ef-811f-e56b675031d8-istio-token\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.113240 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.113226    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-workload-certs\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.113358 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.113340    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-credential-socket\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.113398 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.113378    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-workload-socket\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.113476 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.113455    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d861cce2-0e36-40ef-811f-e56b675031d8-istio-podinfo\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.139407 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.139376    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"]
Oct 30 23:24:26.214392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.214361    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-istio-envoy\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.214525 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.214405    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-istio-data\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.214525 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.214455    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d861cce2-0e36-40ef-811f-e56b675031d8-istiod-ca-cert\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.214525 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.214475    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lr9z\" (UniqueName: \"kubernetes.io/projected/d861cce2-0e36-40ef-811f-e56b675031d8-kube-api-access-2lr9z\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.214737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.214530    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d861cce2-0e36-40ef-811f-e56b675031d8-istio-token\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.214737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.214557    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-workload-certs\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.214737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.214688    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-credential-socket\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.214737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.214724    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-workload-socket\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.214922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.214771    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d861cce2-0e36-40ef-811f-e56b675031d8-istio-podinfo\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.214983 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.214934    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-istio-data\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.215119 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.215088    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-credential-socket\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.215236 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.215120    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-workload-certs\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.215284 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.215247    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d861cce2-0e36-40ef-811f-e56b675031d8-istiod-ca-cert\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.215347 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.215217    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-workload-socket\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.216842 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.216824    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-istio-envoy\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.217177 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.217157    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d861cce2-0e36-40ef-811f-e56b675031d8-istio-podinfo\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.271963 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.271937    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d861cce2-0e36-40ef-811f-e56b675031d8-istio-token\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.272540 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.272516    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lr9z\" (UniqueName: \"kubernetes.io/projected/d861cce2-0e36-40ef-811f-e56b675031d8-kube-api-access-2lr9z\") pod \"e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") " pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.345365 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.345334    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:26.777029 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.777008    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"]
Oct 30 23:24:26.778571 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:24:26.778544    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd861cce2_0e36_40ef_811f_e56b675031d8.slice/crio-2d47bd0f54851a1663017a29a49194e015839418b36ac769a0b86815f474acde WatchSource:0}: Error finding container 2d47bd0f54851a1663017a29a49194e015839418b36ac769a0b86815f474acde: Status 404 returned error can't find the container with id 2d47bd0f54851a1663017a29a49194e015839418b36ac769a0b86815f474acde
Oct 30 23:24:26.780851 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.780824    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:24:26.780917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.780883    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:24:26.780954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.780920    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:24:26.997130 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.997096    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5" event={"ID":"d861cce2-0e36-40ef-811f-e56b675031d8","Type":"ContainerStarted","Data":"3de9a68ebdfb1ee7599f7943a8042a8b2d4d99988360fe06f5020cc888a49a11"}
Oct 30 23:24:26.997335 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:26.997173    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5" event={"ID":"d861cce2-0e36-40ef-811f-e56b675031d8","Type":"ContainerStarted","Data":"2d47bd0f54851a1663017a29a49194e015839418b36ac769a0b86815f474acde"}
Oct 30 23:24:27.134683 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:27.134630    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5" podStartSLOduration=2.1346164930000002 podStartE2EDuration="2.134616493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:24:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:27.131569387 +0000 UTC m=+1053.019475739" watchObservedRunningTime="2026-10-30 23:24:27.134616493 +0000 UTC m=+1053.022522840"
Oct 30 23:24:27.345570 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:27.345534    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:27.347153 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:27.347125    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 23:24:27.347311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:27.347183    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5" podUID="d861cce2-0e36-40ef-811f-e56b675031d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 23:24:28.346480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:28.345997    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 23:24:28.346480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:28.346098    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5" podUID="d861cce2-0e36-40ef-811f-e56b675031d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 23:24:29.346695 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:29.346654    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 23:24:29.347077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:29.346713    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5" podUID="d861cce2-0e36-40ef-811f-e56b675031d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 23:24:30.346200 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:30.346160    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 23:24:30.346357 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:30.346232    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5" podUID="d861cce2-0e36-40ef-811f-e56b675031d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 23:24:31.346342 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:31.346307    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 23:24:31.346795 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:31.346379    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5" podUID="d861cce2-0e36-40ef-811f-e56b675031d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 23:24:31.862645 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:31.857828    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s"]
Oct 30 23:24:31.877665 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:31.872141    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s"
Oct 30 23:24:31.907148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:31.907117    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s"]
Oct 30 23:24:31.909027 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:31.909006    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9961412-5bdf-498c-91ff-97a6534629bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4a128de-7778s\" (UID: \"c9961412-5bdf-498c-91ff-97a6534629bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s"
Oct 30 23:24:31.909194 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:31.909174    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwdrb\" (UniqueName: \"kubernetes.io/projected/c9961412-5bdf-498c-91ff-97a6534629bb-kube-api-access-pwdrb\") pod \"maas-api-revoke-sub-b4a128de-7778s\" (UID: \"c9961412-5bdf-498c-91ff-97a6534629bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s"
Oct 30 23:24:31.909282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:31.909224    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9961412-5bdf-498c-91ff-97a6534629bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4a128de-7778s\" (UID: \"c9961412-5bdf-498c-91ff-97a6534629bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s"
Oct 30 23:24:31.950157 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:31.950127    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:24:31.950300 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:31.950199    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:24:31.950300 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:31.950267    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:24:31.980619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:31.979766    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:24:32.010576 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:32.010549    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwdrb\" (UniqueName: \"kubernetes.io/projected/c9961412-5bdf-498c-91ff-97a6534629bb-kube-api-access-pwdrb\") pod \"maas-api-revoke-sub-b4a128de-7778s\" (UID: \"c9961412-5bdf-498c-91ff-97a6534629bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s"
Oct 30 23:24:32.010706 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:32.010638    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9961412-5bdf-498c-91ff-97a6534629bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4a128de-7778s\" (UID: \"c9961412-5bdf-498c-91ff-97a6534629bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s"
Oct 30 23:24:32.010772 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:32.010704    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9961412-5bdf-498c-91ff-97a6534629bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4a128de-7778s\" (UID: \"c9961412-5bdf-498c-91ff-97a6534629bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s"
Oct 30 23:24:32.011241 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:32.011223    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9961412-5bdf-498c-91ff-97a6534629bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4a128de-7778s\" (UID: \"c9961412-5bdf-498c-91ff-97a6534629bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s"
Oct 30 23:24:32.013154 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:32.013127    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9961412-5bdf-498c-91ff-97a6534629bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4a128de-7778s\" (UID: \"c9961412-5bdf-498c-91ff-97a6534629bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s"
Oct 30 23:24:32.082441 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:32.082409    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwdrb\" (UniqueName: \"kubernetes.io/projected/c9961412-5bdf-498c-91ff-97a6534629bb-kube-api-access-pwdrb\") pod \"maas-api-revoke-sub-b4a128de-7778s\" (UID: \"c9961412-5bdf-498c-91ff-97a6534629bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s"
Oct 30 23:24:32.208947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:32.208856    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s"
Oct 30 23:24:32.346236 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:32.346204    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 23:24:32.346407 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:32.346255    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5" podUID="d861cce2-0e36-40ef-811f-e56b675031d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 23:24:32.607975 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:32.607948    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s"]
Oct 30 23:24:32.608694 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:24:32.608667    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9961412_5bdf_498c_91ff_97a6534629bb.slice/crio-0d7eceb3fcd5c7eb0118a6e9552e316faadf302049dfc839842bc105ad8e9dbd WatchSource:0}: Error finding container 0d7eceb3fcd5c7eb0118a6e9552e316faadf302049dfc839842bc105ad8e9dbd: Status 404 returned error can't find the container with id 0d7eceb3fcd5c7eb0118a6e9552e316faadf302049dfc839842bc105ad8e9dbd
Oct 30 23:24:33.035783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:33.035749    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s" event={"ID":"c9961412-5bdf-498c-91ff-97a6534629bb","Type":"ContainerStarted","Data":"ee8334303a91988aede8266b6442c4e6024f93b4846e5d94cd4fa88c89c08907"}
Oct 30 23:24:33.035953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:33.035811    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s" event={"ID":"c9961412-5bdf-498c-91ff-97a6534629bb","Type":"ContainerStarted","Data":"0d7eceb3fcd5c7eb0118a6e9552e316faadf302049dfc839842bc105ad8e9dbd"}
Oct 30 23:24:33.152009 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:33.151911    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s" podStartSLOduration=2.151898711 podStartE2EDuration="2.151898711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:33.149763784 +0000 UTC m=+1059.037670135" watchObservedRunningTime="2026-10-30 23:24:33.151898711 +0000 UTC m=+1059.039805064"
Oct 30 23:24:34.042828 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:34.042791    5078 generic.go:356] "Generic (PLEG): container finished" podID="c9961412-5bdf-498c-91ff-97a6534629bb" containerID="ee8334303a91988aede8266b6442c4e6024f93b4846e5d94cd4fa88c89c08907" exitCode=0
Oct 30 23:24:34.043235 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:34.042882    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s" event={"ID":"c9961412-5bdf-498c-91ff-97a6534629bb","Type":"ContainerDied","Data":"ee8334303a91988aede8266b6442c4e6024f93b4846e5d94cd4fa88c89c08907"}
Oct 30 23:24:34.205751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:34.205719    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:34.206151 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:34.206128    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:34.212118 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:34.212097    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:24:35.200714 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:35.200687    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s"
Oct 30 23:24:35.241684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:35.241654    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9961412-5bdf-498c-91ff-97a6534629bb-kube-api-access-pwdrb\" (UniqueName: \"kubernetes.io/projected/c9961412-5bdf-498c-91ff-97a6534629bb-kube-api-access-pwdrb\") pod \"c9961412-5bdf-498c-91ff-97a6534629bb\" (UID: \"c9961412-5bdf-498c-91ff-97a6534629bb\") "
Oct 30 23:24:35.241838 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:35.241750    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9961412-5bdf-498c-91ff-97a6534629bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9961412-5bdf-498c-91ff-97a6534629bb-maas-api-xks-ca\") pod \"c9961412-5bdf-498c-91ff-97a6534629bb\" (UID: \"c9961412-5bdf-498c-91ff-97a6534629bb\") "
Oct 30 23:24:35.241838 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:35.241830    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9961412-5bdf-498c-91ff-97a6534629bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9961412-5bdf-498c-91ff-97a6534629bb-maas-api-openshift-ca\") pod \"c9961412-5bdf-498c-91ff-97a6534629bb\" (UID: \"c9961412-5bdf-498c-91ff-97a6534629bb\") "
Oct 30 23:24:35.242180 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:35.242158    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9961412-5bdf-498c-91ff-97a6534629bb-maas-api-openshift-ca" pod "c9961412-5bdf-498c-91ff-97a6534629bb" (UID: "c9961412-5bdf-498c-91ff-97a6534629bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:24:35.243875 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:35.243851    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9961412-5bdf-498c-91ff-97a6534629bb-maas-api-xks-ca" pod "c9961412-5bdf-498c-91ff-97a6534629bb" (UID: "c9961412-5bdf-498c-91ff-97a6534629bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:24:35.243976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:35.243879    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9961412-5bdf-498c-91ff-97a6534629bb-kube-api-access-pwdrb" pod "c9961412-5bdf-498c-91ff-97a6534629bb" (UID: "c9961412-5bdf-498c-91ff-97a6534629bb"). InnerVolumeSpecName "kube-api-access-pwdrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:24:35.343370 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:35.343290    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9961412-5bdf-498c-91ff-97a6534629bb-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:24:35.343370 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:35.343323    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwdrb\" (UniqueName: \"kubernetes.io/projected/c9961412-5bdf-498c-91ff-97a6534629bb-kube-api-access-pwdrb\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:24:35.343370 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:35.343340    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9961412-5bdf-498c-91ff-97a6534629bb-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:24:36.056300 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:36.056268    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s"
Oct 30 23:24:36.056504 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:36.056270    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s" event={"ID":"c9961412-5bdf-498c-91ff-97a6534629bb","Type":"ContainerDied","Data":"0d7eceb3fcd5c7eb0118a6e9552e316faadf302049dfc839842bc105ad8e9dbd"}
Oct 30 23:24:36.056504 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:36.056413    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d7eceb3fcd5c7eb0118a6e9552e316faadf302049dfc839842bc105ad8e9dbd"
Oct 30 23:24:43.947389 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:43.947348    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"]
Oct 30 23:24:43.947981 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:43.947847    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c9961412-5bdf-498c-91ff-97a6534629bb" containerName="revoke-keys"
Oct 30 23:24:43.947981 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:43.947863    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9961412-5bdf-498c-91ff-97a6534629bb" containerName="revoke-keys"
Oct 30 23:24:43.948074 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:43.947982    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9961412-5bdf-498c-91ff-97a6534629bb" containerName="revoke-keys"
Oct 30 23:24:43.951474 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:43.951451    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.007328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.007297    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"]
Oct 30 23:24:44.037305 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.037272    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-tmp-dir\") pod \"distinct-w3-2cfb80-kserve-74c444c669-mxn9z\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") " pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.037474 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.037325    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-home\") pod \"distinct-w3-2cfb80-kserve-74c444c669-mxn9z\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") " pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.037515 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.037488    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-dshm\") pod \"distinct-w3-2cfb80-kserve-74c444c669-mxn9z\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") " pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.037565 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.037549    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvql7\" (UniqueName: \"kubernetes.io/projected/463812b2-d2a2-49c6-96c2-ee32a4745d71-kube-api-access-gvql7\") pod \"distinct-w3-2cfb80-kserve-74c444c669-mxn9z\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") " pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.037626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.037612    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/463812b2-d2a2-49c6-96c2-ee32a4745d71-tls-certs\") pod \"distinct-w3-2cfb80-kserve-74c444c669-mxn9z\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") " pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.037664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.037647    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-model-cache\") pod \"distinct-w3-2cfb80-kserve-74c444c669-mxn9z\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") " pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.053041 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.053017    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-2cfb80\"/\"distinct-w3-2cfb80-kserve-self-signed-certs\""
Oct 30 23:24:44.141625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.138529    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-dshm\") pod \"distinct-w3-2cfb80-kserve-74c444c669-mxn9z\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") " pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.141625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.138636    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvql7\" (UniqueName: \"kubernetes.io/projected/463812b2-d2a2-49c6-96c2-ee32a4745d71-kube-api-access-gvql7\") pod \"distinct-w3-2cfb80-kserve-74c444c669-mxn9z\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") " pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.141625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.138695    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/463812b2-d2a2-49c6-96c2-ee32a4745d71-tls-certs\") pod \"distinct-w3-2cfb80-kserve-74c444c669-mxn9z\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") " pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.141625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.138723    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-model-cache\") pod \"distinct-w3-2cfb80-kserve-74c444c669-mxn9z\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") " pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.141625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.138785    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-tmp-dir\") pod \"distinct-w3-2cfb80-kserve-74c444c669-mxn9z\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") " pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.141625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.138831    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-home\") pod \"distinct-w3-2cfb80-kserve-74c444c669-mxn9z\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") " pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.141625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.139336    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-home\") pod \"distinct-w3-2cfb80-kserve-74c444c669-mxn9z\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") " pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.149443 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.143742    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-model-cache\") pod \"distinct-w3-2cfb80-kserve-74c444c669-mxn9z\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") " pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.149443 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.146929    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-dshm\") pod \"distinct-w3-2cfb80-kserve-74c444c669-mxn9z\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") " pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.149443 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.147156    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-tmp-dir\") pod \"distinct-w3-2cfb80-kserve-74c444c669-mxn9z\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") " pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.149443 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.148561    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/463812b2-d2a2-49c6-96c2-ee32a4745d71-tls-certs\") pod \"distinct-w3-2cfb80-kserve-74c444c669-mxn9z\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") " pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.201081 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.201003    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvql7\" (UniqueName: \"kubernetes.io/projected/463812b2-d2a2-49c6-96c2-ee32a4745d71-kube-api-access-gvql7\") pod \"distinct-w3-2cfb80-kserve-74c444c669-mxn9z\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") " pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.263422 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.263385    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:44.640007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:44.639794    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"]
Oct 30 23:24:44.640990 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:24:44.640962    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod463812b2_d2a2_49c6_96c2_ee32a4745d71.slice/crio-112c499de090d52a4d3e4e451c9bae4b208160a36f5719a6331398ee7d640426 WatchSource:0}: Error finding container 112c499de090d52a4d3e4e451c9bae4b208160a36f5719a6331398ee7d640426: Status 404 returned error can't find the container with id 112c499de090d52a4d3e4e451c9bae4b208160a36f5719a6331398ee7d640426
Oct 30 23:24:45.160475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:45.160445    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" event={"ID":"463812b2-d2a2-49c6-96c2-ee32a4745d71","Type":"ContainerStarted","Data":"2158d3f2ad787f470e208cdcc290ad48d901ca927108cbf198a94f5b624aaa0a"}
Oct 30 23:24:45.160870 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:45.160484    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" event={"ID":"463812b2-d2a2-49c6-96c2-ee32a4745d71","Type":"ContainerStarted","Data":"112c499de090d52a4d3e4e451c9bae4b208160a36f5719a6331398ee7d640426"}
Oct 30 23:24:45.264361 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:45.264320    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:45.275827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:45.275776    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podStartSLOduration=2.275759178 podStartE2EDuration="2.275759178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:24:45.274289745 +0000 UTC m=+1071.162196097" watchObservedRunningTime="2026-10-30 23:24:45.275759178 +0000 UTC m=+1071.163665532"
Oct 30 23:24:47.591365 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:24:47.591326    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:24:54.273836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:54.273798    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:54.345513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:54.345481    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:24:55.252206 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:24:55.252180    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:25:12.428414 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.428380    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"]
Oct 30 23:25:12.432269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.432252    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.569659 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.569624    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-envoy\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.569837 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.569696    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26c4072-f01f-45ae-bb38-4e62b8e87617-istiod-ca-cert\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.569837 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.569737    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-workload-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.569837 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.569759    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x87r5\" (UniqueName: \"kubernetes.io/projected/c26c4072-f01f-45ae-bb38-4e62b8e87617-kube-api-access-x87r5\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.569837 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.569822    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-credential-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.569981 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.569844    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-token\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.569981 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.569882    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-workload-certs\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.570044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.569972    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-data\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.570077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.570048    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-podinfo\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.637804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.637768    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"]
Oct 30 23:25:12.670971 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.670941    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-workload-certs\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.671159 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.670999    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-data\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.671159 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.671031    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-podinfo\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.671159 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.671067    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-envoy\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.671159 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.671119    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26c4072-f01f-45ae-bb38-4e62b8e87617-istiod-ca-cert\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.671389 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.671166    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-workload-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.671389 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.671198    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x87r5\" (UniqueName: \"kubernetes.io/projected/c26c4072-f01f-45ae-bb38-4e62b8e87617-kube-api-access-x87r5\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.671389 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.671239    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-credential-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.671389 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.671272    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-token\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.671631 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.671413    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-workload-certs\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.671631 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.671447    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-data\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.671631 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.671606    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-workload-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.671803 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.671745    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-credential-socket\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.672039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.672018    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26c4072-f01f-45ae-bb38-4e62b8e87617-istiod-ca-cert\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.673430 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.673409    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-envoy\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.673671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.673656    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-podinfo\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.728083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.728011    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-token\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.728207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.728092    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x87r5\" (UniqueName: \"kubernetes.io/projected/c26c4072-f01f-45ae-bb38-4e62b8e87617-kube-api-access-x87r5\") pod \"e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") " pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:12.744820 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:12.744797    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:13.171304 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:13.171276    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"]
Oct 30 23:25:13.171443 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:25:13.171417    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26c4072_f01f_45ae_bb38_4e62b8e87617.slice/crio-cdffdb62ec0919d9c410653dea6a0e08ab9d44fe68d1b5483f599938be1df0dd WatchSource:0}: Error finding container cdffdb62ec0919d9c410653dea6a0e08ab9d44fe68d1b5483f599938be1df0dd: Status 404 returned error can't find the container with id cdffdb62ec0919d9c410653dea6a0e08ab9d44fe68d1b5483f599938be1df0dd
Oct 30 23:25:13.173680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:13.173651    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:25:13.173751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:13.173714    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:25:13.173751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:13.173747    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:25:13.358168 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:13.358131    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch" event={"ID":"c26c4072-f01f-45ae-bb38-4e62b8e87617","Type":"ContainerStarted","Data":"520e91a0e5734fe74c1ec55bb42716604d0e61a0b71d9b7d28edf926e08b7087"}
Oct 30 23:25:13.358368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:13.358193    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch" event={"ID":"c26c4072-f01f-45ae-bb38-4e62b8e87617","Type":"ContainerStarted","Data":"cdffdb62ec0919d9c410653dea6a0e08ab9d44fe68d1b5483f599938be1df0dd"}
Oct 30 23:25:13.481419 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:13.481328    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch" podStartSLOduration=1.481313487 podStartE2EDuration="1.481313487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:25:13.477205272 +0000 UTC m=+1099.365111637" watchObservedRunningTime="2026-10-30 23:25:13.481313487 +0000 UTC m=+1099.369219839"
Oct 30 23:25:13.745683 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:13.745589    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:14.747763 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:14.747731    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 23:25:14.748333 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:14.747791    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch" podUID="c26c4072-f01f-45ae-bb38-4e62b8e87617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 23:25:15.745454 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:15.745417    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 23:25:15.745779 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:15.745476    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch" podUID="c26c4072-f01f-45ae-bb38-4e62b8e87617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 23:25:16.745199 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:16.745162    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 23:25:16.745589 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:16.745219    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch" podUID="c26c4072-f01f-45ae-bb38-4e62b8e87617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 23:25:17.607488 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:25:17.607453    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:25:17.745398 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:17.745364    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 23:25:17.745835 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:17.745420    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch" podUID="c26c4072-f01f-45ae-bb38-4e62b8e87617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 23:25:18.746239 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:18.746202    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 23:25:18.746693 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:18.746263    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch" podUID="c26c4072-f01f-45ae-bb38-4e62b8e87617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 23:25:19.174864 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.174824    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"]
Oct 30 23:25:19.178869 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.178847    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.253227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.253194    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-home\") pod \"distinct-w0-057303-kserve-76dbfb9f6f-4f566\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") " pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.253407 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.253235    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10cebe5f-5bd1-4d37-850f-a76663e0571e-tls-certs\") pod \"distinct-w0-057303-kserve-76dbfb9f6f-4f566\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") " pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.253407 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.253363    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-tmp-dir\") pod \"distinct-w0-057303-kserve-76dbfb9f6f-4f566\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") " pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.253407 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.253402    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-dshm\") pod \"distinct-w0-057303-kserve-76dbfb9f6f-4f566\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") " pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.253547 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.253428    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg7vj\" (UniqueName: \"kubernetes.io/projected/10cebe5f-5bd1-4d37-850f-a76663e0571e-kube-api-access-tg7vj\") pod \"distinct-w0-057303-kserve-76dbfb9f6f-4f566\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") " pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.253587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.253566    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-model-cache\") pod \"distinct-w0-057303-kserve-76dbfb9f6f-4f566\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") " pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.282241 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.282208    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-057303\"/\"distinct-w0-057303-kserve-self-signed-certs\""
Oct 30 23:25:19.285611 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.285572    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"]
Oct 30 23:25:19.359251 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.359201    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-model-cache\") pod \"distinct-w0-057303-kserve-76dbfb9f6f-4f566\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") " pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.360136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.360112    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-home\") pod \"distinct-w0-057303-kserve-76dbfb9f6f-4f566\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") " pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.360359 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.360341    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10cebe5f-5bd1-4d37-850f-a76663e0571e-tls-certs\") pod \"distinct-w0-057303-kserve-76dbfb9f6f-4f566\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") " pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.360543 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.360518    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-tmp-dir\") pod \"distinct-w0-057303-kserve-76dbfb9f6f-4f566\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") " pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.360706 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.360681    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-dshm\") pod \"distinct-w0-057303-kserve-76dbfb9f6f-4f566\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") " pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.360927 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.360909    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg7vj\" (UniqueName: \"kubernetes.io/projected/10cebe5f-5bd1-4d37-850f-a76663e0571e-kube-api-access-tg7vj\") pod \"distinct-w0-057303-kserve-76dbfb9f6f-4f566\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") " pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.361465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.359897    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-model-cache\") pod \"distinct-w0-057303-kserve-76dbfb9f6f-4f566\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") " pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.361905 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.361885    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-home\") pod \"distinct-w0-057303-kserve-76dbfb9f6f-4f566\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") " pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.378911 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.378857    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-tmp-dir\") pod \"distinct-w0-057303-kserve-76dbfb9f6f-4f566\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") " pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.385901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.385794    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-dshm\") pod \"distinct-w0-057303-kserve-76dbfb9f6f-4f566\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") " pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.386950 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.386926    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10cebe5f-5bd1-4d37-850f-a76663e0571e-tls-certs\") pod \"distinct-w0-057303-kserve-76dbfb9f6f-4f566\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") " pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.418381 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.418351    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg7vj\" (UniqueName: \"kubernetes.io/projected/10cebe5f-5bd1-4d37-850f-a76663e0571e-kube-api-access-tg7vj\") pod \"distinct-w0-057303-kserve-76dbfb9f6f-4f566\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") " pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.491239 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.491158    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:19.745436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.745352    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused" start-of-body=
Oct 30 23:25:19.745436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.745403    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch" podUID="c26c4072-f01f-45ae-bb38-4e62b8e87617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.72:15021/healthz/ready\": dial tcp 10.133.0.72:15021: connect: connection refused"
Oct 30 23:25:19.955064 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:19.955037    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"]
Oct 30 23:25:19.956416 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:25:19.956388    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10cebe5f_5bd1_4d37_850f_a76663e0571e.slice/crio-ba2f02af332d4d8a4c95e8a2363944ddf7860b1b29211948287f317d0d340ddc WatchSource:0}: Error finding container ba2f02af332d4d8a4c95e8a2363944ddf7860b1b29211948287f317d0d340ddc: Status 404 returned error can't find the container with id ba2f02af332d4d8a4c95e8a2363944ddf7860b1b29211948287f317d0d340ddc
Oct 30 23:25:20.425248 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:20.425209    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" event={"ID":"10cebe5f-5bd1-4d37-850f-a76663e0571e","Type":"ContainerStarted","Data":"be8a449bd71aa6d8cf193e4af762092bc44f1ab6d0b2aa6ab198d6bfba7d846d"}
Oct 30 23:25:20.426576 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:20.426549    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" event={"ID":"10cebe5f-5bd1-4d37-850f-a76663e0571e","Type":"ContainerStarted","Data":"ba2f02af332d4d8a4c95e8a2363944ddf7860b1b29211948287f317d0d340ddc"}
Oct 30 23:25:20.491314 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:20.491284    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:20.546821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:20.546767    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podStartSLOduration=1.546750445 podStartE2EDuration="1.546750445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:25:20.54291047 +0000 UTC m=+1106.430816823" watchObservedRunningTime="2026-10-30 23:25:20.546750445 +0000 UTC m=+1106.434656801"
Oct 30 23:25:20.750681 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:20.750582    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:20.750921 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:20.750901    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:20.751659 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:20.751640    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:25:24.520900 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.520866    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"]
Oct 30 23:25:24.524685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.524670    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.627730 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.627701    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-5dc3c0\"/\"distinct-w5-5dc3c0-kserve-self-signed-certs\""
Oct 30 23:25:24.630987 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.630963    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"]
Oct 30 23:25:24.635339 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.635312    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-home\") pod \"distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.635463 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.635408    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-dshm\") pod \"distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.635516 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.635468    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7mcp\" (UniqueName: \"kubernetes.io/projected/5bf2b89c-d52c-41bb-a594-4f87590ba512-kube-api-access-p7mcp\") pod \"distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.635516 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.635505    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-model-cache\") pod \"distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.635623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.635568    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-tmp-dir\") pod \"distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.635733 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.635718    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf2b89c-d52c-41bb-a594-4f87590ba512-tls-certs\") pod \"distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.737080 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.737046    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-home\") pod \"distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.737263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.737110    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-dshm\") pod \"distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.737263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.737172    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7mcp\" (UniqueName: \"kubernetes.io/projected/5bf2b89c-d52c-41bb-a594-4f87590ba512-kube-api-access-p7mcp\") pod \"distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.737263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.737204    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-model-cache\") pod \"distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.737263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.737227    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-tmp-dir\") pod \"distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.737453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.737306    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf2b89c-d52c-41bb-a594-4f87590ba512-tls-certs\") pod \"distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.737515 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.737445    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-home\") pod \"distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.737650 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.737632    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-model-cache\") pod \"distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.737745 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.737728    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-tmp-dir\") pod \"distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.739520 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.739501    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-dshm\") pod \"distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.739946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.739926    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf2b89c-d52c-41bb-a594-4f87590ba512-tls-certs\") pod \"distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.797904 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.797873    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7mcp\" (UniqueName: \"kubernetes.io/projected/5bf2b89c-d52c-41bb-a594-4f87590ba512-kube-api-access-p7mcp\") pod \"distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") " pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:24.836632 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:24.836582    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:25.239399 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:25.239368    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"]
Oct 30 23:25:25.240157 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:25:25.240125    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bf2b89c_d52c_41bb_a594_4f87590ba512.slice/crio-c81dd4b3d6a721c70e2359ab625133ffca22e95ed734e492fcae866ea21bdcf1 WatchSource:0}: Error finding container c81dd4b3d6a721c70e2359ab625133ffca22e95ed734e492fcae866ea21bdcf1: Status 404 returned error can't find the container with id c81dd4b3d6a721c70e2359ab625133ffca22e95ed734e492fcae866ea21bdcf1
Oct 30 23:25:25.457747 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:25.457714    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" event={"ID":"5bf2b89c-d52c-41bb-a594-4f87590ba512","Type":"ContainerStarted","Data":"4d143441d651e9e12883eaea28707b319726b018a9b14f476e90f8ee95885d62"}
Oct 30 23:25:25.457898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:25.457759    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" event={"ID":"5bf2b89c-d52c-41bb-a594-4f87590ba512","Type":"ContainerStarted","Data":"c81dd4b3d6a721c70e2359ab625133ffca22e95ed734e492fcae866ea21bdcf1"}
Oct 30 23:25:25.576264 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:25.576209    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podStartSLOduration=1.5761948220000002 podStartE2EDuration="1.576194822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:25:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:25:25.575077606 +0000 UTC m=+1111.462983961" watchObservedRunningTime="2026-10-30 23:25:25.576194822 +0000 UTC m=+1111.464101174"
Oct 30 23:25:25.837215 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:25.837131    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:29.492108 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:29.492079    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:29.504469 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:29.504443    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:30.502940 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:30.502905    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:25:34.837624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:34.837563    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:34.850188 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:34.850156    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:35.543523 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:25:35.543484    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:25:47.591377 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:25:47.591338    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:26:12.536944 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.536896    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"]
Oct 30 23:26:12.540678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.540661    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.635415 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.635388    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-podinfo\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.635546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.635431    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-workload-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.635618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.635548    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-workload-certs\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.635662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.635630    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkhz4\" (UniqueName: \"kubernetes.io/projected/6859c991-1ae2-4055-8d25-5b0d1e858d4a-kube-api-access-tkhz4\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.635700 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.635684    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-envoy\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.635734 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.635699    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-data\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.635854 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.635840    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-credential-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.635890 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.635883    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-token\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.635920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.635910    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istiod-ca-cert\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.645628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.645585    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"]
Oct 30 23:26:12.737393 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.737362    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-credential-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.737546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.737417    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-token\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.737546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.737533    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istiod-ca-cert\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.737658 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.737579    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-podinfo\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.737696 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.737662    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-workload-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.737746 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.737731    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-workload-certs\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.737798 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.737784    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkhz4\" (UniqueName: \"kubernetes.io/projected/6859c991-1ae2-4055-8d25-5b0d1e858d4a-kube-api-access-tkhz4\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.737840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.737803    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-credential-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.737963 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.737940    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-envoy\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.738033 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.737968    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-workload-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.738033 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.737976    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-data\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.738145 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.738034    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-workload-certs\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.738228 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.738205    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istiod-ca-cert\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.738301 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.738279    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-data\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.740175 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.740159    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-podinfo\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.740230 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.740176    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-envoy\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.794094 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.794065    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-token\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.794781 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.794761    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkhz4\" (UniqueName: \"kubernetes.io/projected/6859c991-1ae2-4055-8d25-5b0d1e858d4a-kube-api-access-tkhz4\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:12.853385 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:12.853352    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:13.280347 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:13.280274    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"]
Oct 30 23:26:13.319149 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:26:13.319123    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6859c991_1ae2_4055_8d25_5b0d1e858d4a.slice/crio-82997942111b3b9c601b9e3bd1998760548b74fb7063158e764c25b289fdf414 WatchSource:0}: Error finding container 82997942111b3b9c601b9e3bd1998760548b74fb7063158e764c25b289fdf414: Status 404 returned error can't find the container with id 82997942111b3b9c601b9e3bd1998760548b74fb7063158e764c25b289fdf414
Oct 30 23:26:13.321534 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:13.321501    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:26:13.321653 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:13.321576    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:26:13.321653 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:13.321639    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:26:13.821192 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:13.821162    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r" event={"ID":"6859c991-1ae2-4055-8d25-5b0d1e858d4a","Type":"ContainerStarted","Data":"bfbfb43789a1689f912ae423645fa6a2b3a8eecc0e31e46edb01f13459e14de4"}
Oct 30 23:26:13.821638 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:13.821211    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r" event={"ID":"6859c991-1ae2-4055-8d25-5b0d1e858d4a","Type":"ContainerStarted","Data":"82997942111b3b9c601b9e3bd1998760548b74fb7063158e764c25b289fdf414"}
Oct 30 23:26:13.854120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:13.854096    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:13.939312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:13.939261    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r" podStartSLOduration=1.939246962 podStartE2EDuration="1.939246962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:26:13.937507835 +0000 UTC m=+1159.825414188" watchObservedRunningTime="2026-10-30 23:26:13.939246962 +0000 UTC m=+1159.827153315"
Oct 30 23:26:14.855635 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:14.855576    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:26:14.856081 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:14.855662    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r" podUID="6859c991-1ae2-4055-8d25-5b0d1e858d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 23:26:15.854558 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:15.854528    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:26:15.854748 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:15.854577    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r" podUID="6859c991-1ae2-4055-8d25-5b0d1e858d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 23:26:16.854482 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:16.854442    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:26:16.854916 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:16.854505    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r" podUID="6859c991-1ae2-4055-8d25-5b0d1e858d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 23:26:17.613293 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:26:17.613254    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:26:17.854044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:17.854010    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:26:17.854223 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:17.854067    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r" podUID="6859c991-1ae2-4055-8d25-5b0d1e858d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 23:26:18.853659 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:18.853624    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:26:18.854206 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:18.853712    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r" podUID="6859c991-1ae2-4055-8d25-5b0d1e858d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 23:26:19.854115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:19.854084    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused" start-of-body=
Oct 30 23:26:19.854573 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:19.854141    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r" podUID="6859c991-1ae2-4055-8d25-5b0d1e858d4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.75:15021/healthz/ready\": dial tcp 10.133.0.75:15021: connect: connection refused"
Oct 30 23:26:20.981436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:20.981398    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:20.981946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:20.981871    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:20.982808 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:20.982787    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:26:27.852811 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:27.852778    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-fbxsv-m7vqv"]
Oct 30 23:26:27.856878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:27.856860    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fbxsv-m7vqv"
Oct 30 23:26:27.912195 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:27.912169    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fbxsv-m7vqv"]
Oct 30 23:26:27.933353 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:27.933325    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj98z\" (UniqueName: \"kubernetes.io/projected/5465b09b-ab0c-462e-9e44-c3f884253bb6-kube-api-access-kj98z\") pod \"periodic-gathering-fbxsv-m7vqv\" (UID: \"5465b09b-ab0c-462e-9e44-c3f884253bb6\") " pod="openshift-insights/periodic-gathering-fbxsv-m7vqv"
Oct 30 23:26:27.933514 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:27.933378    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5465b09b-ab0c-462e-9e44-c3f884253bb6-trusted-ca-bundle\") pod \"periodic-gathering-fbxsv-m7vqv\" (UID: \"5465b09b-ab0c-462e-9e44-c3f884253bb6\") " pod="openshift-insights/periodic-gathering-fbxsv-m7vqv"
Oct 30 23:26:27.933514 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:27.933408    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5465b09b-ab0c-462e-9e44-c3f884253bb6-service-ca-bundle\") pod \"periodic-gathering-fbxsv-m7vqv\" (UID: \"5465b09b-ab0c-462e-9e44-c3f884253bb6\") " pod="openshift-insights/periodic-gathering-fbxsv-m7vqv"
Oct 30 23:26:27.933644 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:27.933517    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5465b09b-ab0c-462e-9e44-c3f884253bb6-archives-path\") pod \"periodic-gathering-fbxsv-m7vqv\" (UID: \"5465b09b-ab0c-462e-9e44-c3f884253bb6\") " pod="openshift-insights/periodic-gathering-fbxsv-m7vqv"
Oct 30 23:26:27.958089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:27.958062    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6m9zb\""
Oct 30 23:26:27.959184 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:27.959165    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 23:26:28.034924 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.034880    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj98z\" (UniqueName: \"kubernetes.io/projected/5465b09b-ab0c-462e-9e44-c3f884253bb6-kube-api-access-kj98z\") pod \"periodic-gathering-fbxsv-m7vqv\" (UID: \"5465b09b-ab0c-462e-9e44-c3f884253bb6\") " pod="openshift-insights/periodic-gathering-fbxsv-m7vqv"
Oct 30 23:26:28.035119 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.034948    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5465b09b-ab0c-462e-9e44-c3f884253bb6-trusted-ca-bundle\") pod \"periodic-gathering-fbxsv-m7vqv\" (UID: \"5465b09b-ab0c-462e-9e44-c3f884253bb6\") " pod="openshift-insights/periodic-gathering-fbxsv-m7vqv"
Oct 30 23:26:28.035119 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.034975    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5465b09b-ab0c-462e-9e44-c3f884253bb6-service-ca-bundle\") pod \"periodic-gathering-fbxsv-m7vqv\" (UID: \"5465b09b-ab0c-462e-9e44-c3f884253bb6\") " pod="openshift-insights/periodic-gathering-fbxsv-m7vqv"
Oct 30 23:26:28.035119 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.035028    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5465b09b-ab0c-462e-9e44-c3f884253bb6-archives-path\") pod \"periodic-gathering-fbxsv-m7vqv\" (UID: \"5465b09b-ab0c-462e-9e44-c3f884253bb6\") " pod="openshift-insights/periodic-gathering-fbxsv-m7vqv"
Oct 30 23:26:28.035445 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.035422    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5465b09b-ab0c-462e-9e44-c3f884253bb6-archives-path\") pod \"periodic-gathering-fbxsv-m7vqv\" (UID: \"5465b09b-ab0c-462e-9e44-c3f884253bb6\") " pod="openshift-insights/periodic-gathering-fbxsv-m7vqv"
Oct 30 23:26:28.035638 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.035617    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5465b09b-ab0c-462e-9e44-c3f884253bb6-service-ca-bundle\") pod \"periodic-gathering-fbxsv-m7vqv\" (UID: \"5465b09b-ab0c-462e-9e44-c3f884253bb6\") " pod="openshift-insights/periodic-gathering-fbxsv-m7vqv"
Oct 30 23:26:28.058286 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.058255    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 23:26:28.065819 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.065789    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5465b09b-ab0c-462e-9e44-c3f884253bb6-trusted-ca-bundle\") pod \"periodic-gathering-fbxsv-m7vqv\" (UID: \"5465b09b-ab0c-462e-9e44-c3f884253bb6\") " pod="openshift-insights/periodic-gathering-fbxsv-m7vqv"
Oct 30 23:26:28.091973 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.091948    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj98z\" (UniqueName: \"kubernetes.io/projected/5465b09b-ab0c-462e-9e44-c3f884253bb6-kube-api-access-kj98z\") pod \"periodic-gathering-fbxsv-m7vqv\" (UID: \"5465b09b-ab0c-462e-9e44-c3f884253bb6\") " pod="openshift-insights/periodic-gathering-fbxsv-m7vqv"
Oct 30 23:26:28.169734 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.169660    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fbxsv-m7vqv"
Oct 30 23:26:28.546673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.546648    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-fbxsv-m7vqv"]
Oct 30 23:26:28.547674 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:26:28.547648    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5465b09b_ab0c_462e_9e44_c3f884253bb6.slice/crio-de3d0fe4ade8911955b2c8344ff8d7b0e9015c414be7117cdc381c8258fcbf28 WatchSource:0}: Error finding container de3d0fe4ade8911955b2c8344ff8d7b0e9015c414be7117cdc381c8258fcbf28: Status 404 returned error can't find the container with id de3d0fe4ade8911955b2c8344ff8d7b0e9015c414be7117cdc381c8258fcbf28
Oct 30 23:26:28.624671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.624645    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"]
Oct 30 23:26:28.628552 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.628537    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.684724 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.684693    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-workload-certs\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.684905 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.684731    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qct4t\" (UniqueName: \"kubernetes.io/projected/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-kube-api-access-qct4t\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.684905 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.684753    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-data\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.685028 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.684945    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-token\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.685068 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.685039    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-workload-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.685112 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.685071    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-envoy\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.685151 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.685126    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-podinfo\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.685257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.685236    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-credential-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.685296 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.685273    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istiod-ca-cert\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.729396 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.729360    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"]
Oct 30 23:26:28.785891 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.785864    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-workload-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.786089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.785898    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-envoy\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.786089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.785929    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-podinfo\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.786089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.785994    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-credential-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.786089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.786015    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istiod-ca-cert\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.786089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.786034    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-workload-certs\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.786089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.786059    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qct4t\" (UniqueName: \"kubernetes.io/projected/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-kube-api-access-qct4t\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.786089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.786089    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-data\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.786457 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.786230    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-token\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.786457 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.786331    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-workload-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.786571 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.786458    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-workload-certs\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.786571 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.786555    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-data\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.786725 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.786588    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-credential-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.787086 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.787060    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istiod-ca-cert\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.789021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.789002    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-envoy\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.789358 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.789336    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-podinfo\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.843702 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.843634    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-token\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.843812 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.843791    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qct4t\" (UniqueName: \"kubernetes.io/projected/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-kube-api-access-qct4t\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:28.913464 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.913428    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fbxsv-m7vqv" event={"ID":"5465b09b-ab0c-462e-9e44-c3f884253bb6","Type":"ContainerStarted","Data":"19a4ea1a3c4d7c6ed1afd181c6dda344a29c3310a4e4fe15363f1c95b922945d"}
Oct 30 23:26:28.913827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.913485    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fbxsv-m7vqv" event={"ID":"5465b09b-ab0c-462e-9e44-c3f884253bb6","Type":"ContainerStarted","Data":"de3d0fe4ade8911955b2c8344ff8d7b0e9015c414be7117cdc381c8258fcbf28"}
Oct 30 23:26:28.945780 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:28.945754    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:29.029441 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:29.029392    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-fbxsv-m7vqv" podStartSLOduration=2.029378736 podStartE2EDuration="2.029378736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:26:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:26:29.026998796 +0000 UTC m=+1174.914905149" watchObservedRunningTime="2026-10-30 23:26:29.029378736 +0000 UTC m=+1174.917285088"
Oct 30 23:26:29.378777 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:29.378031    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"]
Oct 30 23:26:29.381678 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:26:29.381647    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb7cc10_970b_4a76_9f24_9151f8c7c0c8.slice/crio-0c0bcadbfb269e75b2cf327aaffe47420ec37446cbb95133b3bb05c7b8576cae WatchSource:0}: Error finding container 0c0bcadbfb269e75b2cf327aaffe47420ec37446cbb95133b3bb05c7b8576cae: Status 404 returned error can't find the container with id 0c0bcadbfb269e75b2cf327aaffe47420ec37446cbb95133b3bb05c7b8576cae
Oct 30 23:26:29.384572 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:29.384541    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:26:29.384692 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:29.384632    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:26:29.384692 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:29.384661    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:26:29.920561 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:29.920523    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg" event={"ID":"beb7cc10-970b-4a76-9f24-9151f8c7c0c8","Type":"ContainerStarted","Data":"020f0bbad6d3610327973300fc5470188ec173d9f55015e27dfe19e214ec4ef1"}
Oct 30 23:26:29.920998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:29.920588    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg" event={"ID":"beb7cc10-970b-4a76-9f24-9151f8c7c0c8","Type":"ContainerStarted","Data":"0c0bcadbfb269e75b2cf327aaffe47420ec37446cbb95133b3bb05c7b8576cae"}
Oct 30 23:26:29.946568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:29.946545    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:30.043155 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.043106    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg" podStartSLOduration=2.043092656 podStartE2EDuration="2.043092656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:26:30.040733309 +0000 UTC m=+1175.928639687" watchObservedRunningTime="2026-10-30 23:26:30.043092656 +0000 UTC m=+1175.930999009"
Oct 30 23:26:30.642288 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.642259    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f"]
Oct 30 23:26:30.646379 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.646358    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f"
Oct 30 23:26:30.698680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.698646    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:26:30.707863 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.707834    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10a5cd9-f28f-401e-8199-7a5c022d26b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e4b5de4-f977f\" (UID: \"f10a5cd9-f28f-401e-8199-7a5c022d26b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f"
Oct 30 23:26:30.707990 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.707953    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10a5cd9-f28f-401e-8199-7a5c022d26b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e4b5de4-f977f\" (UID: \"f10a5cd9-f28f-401e-8199-7a5c022d26b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f"
Oct 30 23:26:30.708030 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.707995    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzhgz\" (UniqueName: \"kubernetes.io/projected/f10a5cd9-f28f-401e-8199-7a5c022d26b3-kube-api-access-fzhgz\") pod \"maas-api-revoke-sub-3e4b5de4-f977f\" (UID: \"f10a5cd9-f28f-401e-8199-7a5c022d26b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f"
Oct 30 23:26:30.748050 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.748020    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:26:30.748219 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.748097    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:26:30.748321 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.748033    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:26:30.751719 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.751698    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f"]
Oct 30 23:26:30.809285 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.809253    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10a5cd9-f28f-401e-8199-7a5c022d26b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e4b5de4-f977f\" (UID: \"f10a5cd9-f28f-401e-8199-7a5c022d26b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f"
Oct 30 23:26:30.809452 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.809328    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10a5cd9-f28f-401e-8199-7a5c022d26b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e4b5de4-f977f\" (UID: \"f10a5cd9-f28f-401e-8199-7a5c022d26b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f"
Oct 30 23:26:30.809452 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.809359    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzhgz\" (UniqueName: \"kubernetes.io/projected/f10a5cd9-f28f-401e-8199-7a5c022d26b3-kube-api-access-fzhgz\") pod \"maas-api-revoke-sub-3e4b5de4-f977f\" (UID: \"f10a5cd9-f28f-401e-8199-7a5c022d26b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f"
Oct 30 23:26:30.809967 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.809947    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10a5cd9-f28f-401e-8199-7a5c022d26b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e4b5de4-f977f\" (UID: \"f10a5cd9-f28f-401e-8199-7a5c022d26b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f"
Oct 30 23:26:30.811796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.811777    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10a5cd9-f28f-401e-8199-7a5c022d26b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e4b5de4-f977f\" (UID: \"f10a5cd9-f28f-401e-8199-7a5c022d26b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f"
Oct 30 23:26:30.882662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.882630    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzhgz\" (UniqueName: \"kubernetes.io/projected/f10a5cd9-f28f-401e-8199-7a5c022d26b3-kube-api-access-fzhgz\") pod \"maas-api-revoke-sub-3e4b5de4-f977f\" (UID: \"f10a5cd9-f28f-401e-8199-7a5c022d26b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f"
Oct 30 23:26:30.947550 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.947472    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 23:26:30.947550 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.947521    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg" podUID="beb7cc10-970b-4a76-9f24-9151f8c7c0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 23:26:30.985358 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:30.985320    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f"
Oct 30 23:26:31.381957 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:31.381912    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f"]
Oct 30 23:26:31.384046 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:26:31.384013    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10a5cd9_f28f_401e_8199_7a5c022d26b3.slice/crio-3349cbcd3a2d17c724c113e40be576732319f73ffcea4439f57f5ac202d91ba7 WatchSource:0}: Error finding container 3349cbcd3a2d17c724c113e40be576732319f73ffcea4439f57f5ac202d91ba7: Status 404 returned error can't find the container with id 3349cbcd3a2d17c724c113e40be576732319f73ffcea4439f57f5ac202d91ba7
Oct 30 23:26:31.939013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:31.938976    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f" event={"ID":"f10a5cd9-f28f-401e-8199-7a5c022d26b3","Type":"ContainerStarted","Data":"b914aa5183530f6f654abaa599ba97537e490aa36f81db25bc2073a669faffec"}
Oct 30 23:26:31.939180 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:31.939039    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f" event={"ID":"f10a5cd9-f28f-401e-8199-7a5c022d26b3","Type":"ContainerStarted","Data":"3349cbcd3a2d17c724c113e40be576732319f73ffcea4439f57f5ac202d91ba7"}
Oct 30 23:26:31.946793 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:31.946765    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 23:26:31.946910 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:31.946816    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg" podUID="beb7cc10-970b-4a76-9f24-9151f8c7c0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 23:26:32.052226 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:32.052178    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f" podStartSLOduration=2.052163799 podStartE2EDuration="2.052163799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:26:32.050784897 +0000 UTC m=+1177.938691250" watchObservedRunningTime="2026-10-30 23:26:32.052163799 +0000 UTC m=+1177.940070192"
Oct 30 23:26:32.946082 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:32.946047    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 23:26:32.946262 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:32.946094    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg" podUID="beb7cc10-970b-4a76-9f24-9151f8c7c0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 23:26:32.946809 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:32.946784    5078 generic.go:356] "Generic (PLEG): container finished" podID="f10a5cd9-f28f-401e-8199-7a5c022d26b3" containerID="b914aa5183530f6f654abaa599ba97537e490aa36f81db25bc2073a669faffec" exitCode=0
Oct 30 23:26:32.946909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:32.946851    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f" event={"ID":"f10a5cd9-f28f-401e-8199-7a5c022d26b3","Type":"ContainerDied","Data":"b914aa5183530f6f654abaa599ba97537e490aa36f81db25bc2073a669faffec"}
Oct 30 23:26:33.946401 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:33.946360    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 23:26:33.946890 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:33.946434    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg" podUID="beb7cc10-970b-4a76-9f24-9151f8c7c0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 23:26:34.154532 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:34.154506    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f"
Oct 30 23:26:34.252129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:34.252054    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10a5cd9-f28f-401e-8199-7a5c022d26b3-kube-api-access-fzhgz\" (UniqueName: \"kubernetes.io/projected/f10a5cd9-f28f-401e-8199-7a5c022d26b3-kube-api-access-fzhgz\") pod \"f10a5cd9-f28f-401e-8199-7a5c022d26b3\" (UID: \"f10a5cd9-f28f-401e-8199-7a5c022d26b3\") "
Oct 30 23:26:34.252129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:34.252101    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10a5cd9-f28f-401e-8199-7a5c022d26b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10a5cd9-f28f-401e-8199-7a5c022d26b3-maas-api-openshift-ca\") pod \"f10a5cd9-f28f-401e-8199-7a5c022d26b3\" (UID: \"f10a5cd9-f28f-401e-8199-7a5c022d26b3\") "
Oct 30 23:26:34.252307 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:34.252244    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f10a5cd9-f28f-401e-8199-7a5c022d26b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10a5cd9-f28f-401e-8199-7a5c022d26b3-maas-api-xks-ca\") pod \"f10a5cd9-f28f-401e-8199-7a5c022d26b3\" (UID: \"f10a5cd9-f28f-401e-8199-7a5c022d26b3\") "
Oct 30 23:26:34.252550 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:34.252518    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10a5cd9-f28f-401e-8199-7a5c022d26b3-maas-api-openshift-ca" pod "f10a5cd9-f28f-401e-8199-7a5c022d26b3" (UID: "f10a5cd9-f28f-401e-8199-7a5c022d26b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:26:34.252719 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:34.252704    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10a5cd9-f28f-401e-8199-7a5c022d26b3-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:26:34.254404 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:34.254378    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f10a5cd9-f28f-401e-8199-7a5c022d26b3-maas-api-xks-ca" pod "f10a5cd9-f28f-401e-8199-7a5c022d26b3" (UID: "f10a5cd9-f28f-401e-8199-7a5c022d26b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:26:34.254486 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:34.254409    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10a5cd9-f28f-401e-8199-7a5c022d26b3-kube-api-access-fzhgz" pod "f10a5cd9-f28f-401e-8199-7a5c022d26b3" (UID: "f10a5cd9-f28f-401e-8199-7a5c022d26b3"). InnerVolumeSpecName "kube-api-access-fzhgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:26:34.354108 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:34.354041    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10a5cd9-f28f-401e-8199-7a5c022d26b3-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:26:34.354108 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:34.354072    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzhgz\" (UniqueName: \"kubernetes.io/projected/f10a5cd9-f28f-401e-8199-7a5c022d26b3-kube-api-access-fzhgz\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:26:34.946375 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:34.946338    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 23:26:34.946553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:34.946395    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg" podUID="beb7cc10-970b-4a76-9f24-9151f8c7c0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 23:26:34.984650 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:34.984627    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f"
Oct 30 23:26:34.984798 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:34.984661    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f" event={"ID":"f10a5cd9-f28f-401e-8199-7a5c022d26b3","Type":"ContainerDied","Data":"3349cbcd3a2d17c724c113e40be576732319f73ffcea4439f57f5ac202d91ba7"}
Oct 30 23:26:34.984798 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:34.984731    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3349cbcd3a2d17c724c113e40be576732319f73ffcea4439f57f5ac202d91ba7"
Oct 30 23:26:35.946105 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:35.946066    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 23:26:35.946303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:35.946133    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg" podUID="beb7cc10-970b-4a76-9f24-9151f8c7c0c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 23:26:36.951231 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:36.951199    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:36.951740 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:36.951531    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:36.952281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:36.952260    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:26:45.263904 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.263865    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh"]
Oct 30 23:26:45.264542 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.264521    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f10a5cd9-f28f-401e-8199-7a5c022d26b3" containerName="revoke-keys"
Oct 30 23:26:45.264674 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.264545    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10a5cd9-f28f-401e-8199-7a5c022d26b3" containerName="revoke-keys"
Oct 30 23:26:45.264784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.264769    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10a5cd9-f28f-401e-8199-7a5c022d26b3" containerName="revoke-keys"
Oct 30 23:26:45.268259 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.268238    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh"
Oct 30 23:26:45.324169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.324116    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh"]
Oct 30 23:26:45.369307 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.369286    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:26:45.369455 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.369371    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:26:45.369455 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.369387    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:26:45.370183 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.370167    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:26:45.418736 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.418706    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03b6209-bb59-48eb-9521-0e7a65796e65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a142cf1c-sr5gh\" (UID: \"a03b6209-bb59-48eb-9521-0e7a65796e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh"
Oct 30 23:26:45.418903 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.418750    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w6hw\" (UniqueName: \"kubernetes.io/projected/a03b6209-bb59-48eb-9521-0e7a65796e65-kube-api-access-4w6hw\") pod \"maas-api-revoke-sub-a142cf1c-sr5gh\" (UID: \"a03b6209-bb59-48eb-9521-0e7a65796e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh"
Oct 30 23:26:45.419028 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.418990    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03b6209-bb59-48eb-9521-0e7a65796e65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a142cf1c-sr5gh\" (UID: \"a03b6209-bb59-48eb-9521-0e7a65796e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh"
Oct 30 23:26:45.520170 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.520105    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03b6209-bb59-48eb-9521-0e7a65796e65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a142cf1c-sr5gh\" (UID: \"a03b6209-bb59-48eb-9521-0e7a65796e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh"
Oct 30 23:26:45.520170 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.520148    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w6hw\" (UniqueName: \"kubernetes.io/projected/a03b6209-bb59-48eb-9521-0e7a65796e65-kube-api-access-4w6hw\") pod \"maas-api-revoke-sub-a142cf1c-sr5gh\" (UID: \"a03b6209-bb59-48eb-9521-0e7a65796e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh"
Oct 30 23:26:45.520330 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.520231    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03b6209-bb59-48eb-9521-0e7a65796e65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a142cf1c-sr5gh\" (UID: \"a03b6209-bb59-48eb-9521-0e7a65796e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh"
Oct 30 23:26:45.520806 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.520788    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03b6209-bb59-48eb-9521-0e7a65796e65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a142cf1c-sr5gh\" (UID: \"a03b6209-bb59-48eb-9521-0e7a65796e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh"
Oct 30 23:26:45.522730 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.522708    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03b6209-bb59-48eb-9521-0e7a65796e65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a142cf1c-sr5gh\" (UID: \"a03b6209-bb59-48eb-9521-0e7a65796e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh"
Oct 30 23:26:45.577013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.576984    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w6hw\" (UniqueName: \"kubernetes.io/projected/a03b6209-bb59-48eb-9521-0e7a65796e65-kube-api-access-4w6hw\") pod \"maas-api-revoke-sub-a142cf1c-sr5gh\" (UID: \"a03b6209-bb59-48eb-9521-0e7a65796e65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh"
Oct 30 23:26:45.581836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.581814    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh"
Oct 30 23:26:45.960037 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:45.960008    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh"]
Oct 30 23:26:45.960245 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:26:45.960217    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda03b6209_bb59_48eb_9521_0e7a65796e65.slice/crio-f314f70f36c9945a605d33cfb04a129218a38e04554e90f88b2022b20bdabed7 WatchSource:0}: Error finding container f314f70f36c9945a605d33cfb04a129218a38e04554e90f88b2022b20bdabed7: Status 404 returned error can't find the container with id f314f70f36c9945a605d33cfb04a129218a38e04554e90f88b2022b20bdabed7
Oct 30 23:26:46.058280 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:46.058251    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh" event={"ID":"a03b6209-bb59-48eb-9521-0e7a65796e65","Type":"ContainerStarted","Data":"f314f70f36c9945a605d33cfb04a129218a38e04554e90f88b2022b20bdabed7"}
Oct 30 23:26:47.066127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:47.066089    5078 generic.go:356] "Generic (PLEG): container finished" podID="a03b6209-bb59-48eb-9521-0e7a65796e65" containerID="ba0434a839ec161f63ffc457008cb6f6ce64737dbee375bd777f76a447b14047" exitCode=0
Oct 30 23:26:47.066517 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:47.066134    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh" event={"ID":"a03b6209-bb59-48eb-9521-0e7a65796e65","Type":"ContainerDied","Data":"ba0434a839ec161f63ffc457008cb6f6ce64737dbee375bd777f76a447b14047"}
Oct 30 23:26:47.591133 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:26:47.591099    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:26:48.218878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:48.218855    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh"
Oct 30 23:26:48.353455 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:48.353380    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a03b6209-bb59-48eb-9521-0e7a65796e65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03b6209-bb59-48eb-9521-0e7a65796e65-maas-api-xks-ca\") pod \"a03b6209-bb59-48eb-9521-0e7a65796e65\" (UID: \"a03b6209-bb59-48eb-9521-0e7a65796e65\") "
Oct 30 23:26:48.353455 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:48.353435    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a03b6209-bb59-48eb-9521-0e7a65796e65-kube-api-access-4w6hw\" (UniqueName: \"kubernetes.io/projected/a03b6209-bb59-48eb-9521-0e7a65796e65-kube-api-access-4w6hw\") pod \"a03b6209-bb59-48eb-9521-0e7a65796e65\" (UID: \"a03b6209-bb59-48eb-9521-0e7a65796e65\") "
Oct 30 23:26:48.353673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:48.353530    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a03b6209-bb59-48eb-9521-0e7a65796e65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03b6209-bb59-48eb-9521-0e7a65796e65-maas-api-openshift-ca\") pod \"a03b6209-bb59-48eb-9521-0e7a65796e65\" (UID: \"a03b6209-bb59-48eb-9521-0e7a65796e65\") "
Oct 30 23:26:48.353949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:48.353917    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a03b6209-bb59-48eb-9521-0e7a65796e65-maas-api-openshift-ca" pod "a03b6209-bb59-48eb-9521-0e7a65796e65" (UID: "a03b6209-bb59-48eb-9521-0e7a65796e65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:26:48.354234 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:48.354114    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a03b6209-bb59-48eb-9521-0e7a65796e65-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:26:48.355647 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:48.355622    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a03b6209-bb59-48eb-9521-0e7a65796e65-maas-api-xks-ca" pod "a03b6209-bb59-48eb-9521-0e7a65796e65" (UID: "a03b6209-bb59-48eb-9521-0e7a65796e65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:26:48.355753 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:48.355732    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a03b6209-bb59-48eb-9521-0e7a65796e65-kube-api-access-4w6hw" pod "a03b6209-bb59-48eb-9521-0e7a65796e65" (UID: "a03b6209-bb59-48eb-9521-0e7a65796e65"). InnerVolumeSpecName "kube-api-access-4w6hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:26:48.455709 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:48.455676    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a03b6209-bb59-48eb-9521-0e7a65796e65-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:26:48.455709 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:48.455711    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w6hw\" (UniqueName: \"kubernetes.io/projected/a03b6209-bb59-48eb-9521-0e7a65796e65-kube-api-access-4w6hw\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:26:49.080114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:49.080079    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh" event={"ID":"a03b6209-bb59-48eb-9521-0e7a65796e65","Type":"ContainerDied","Data":"f314f70f36c9945a605d33cfb04a129218a38e04554e90f88b2022b20bdabed7"}
Oct 30 23:26:49.080287 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:49.080134    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f314f70f36c9945a605d33cfb04a129218a38e04554e90f88b2022b20bdabed7"
Oct 30 23:26:49.080287 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:26:49.080086    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh"
Oct 30 23:27:02.992957 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:02.992924    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"]
Oct 30 23:27:02.993452 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:02.993439    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a03b6209-bb59-48eb-9521-0e7a65796e65" containerName="revoke-keys"
Oct 30 23:27:02.993499 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:02.993453    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a03b6209-bb59-48eb-9521-0e7a65796e65" containerName="revoke-keys"
Oct 30 23:27:02.993543 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:02.993534    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a03b6209-bb59-48eb-9521-0e7a65796e65" containerName="revoke-keys"
Oct 30 23:27:02.998529 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:02.998513    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.054428 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.054400    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"]
Oct 30 23:27:03.062514 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.062475    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-dshm\") pod \"distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.062705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.062520    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-model-cache\") pod \"distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.062790 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.062707    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-tls-certs\") pod \"distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.062902 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.062875    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-tmp-dir\") pod \"distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.062972 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.062957    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdgr9\" (UniqueName: \"kubernetes.io/projected/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-kube-api-access-kdgr9\") pod \"distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.063031 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.062999    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-home\") pod \"distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.099349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.099320    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-989a4f\"/\"default-dockercfg-sr59g\""
Oct 30 23:27:03.099513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.099407    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-989a4f\"/\"distinct-2-w4-989a4f-kserve-self-signed-certs\""
Oct 30 23:27:03.100337 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.100318    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-989a4f\"/\"openshift-service-ca.crt\""
Oct 30 23:27:03.100442 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.100394    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-989a4f\"/\"kube-root-ca.crt\""
Oct 30 23:27:03.167669 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.167573    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-model-cache\") pod \"distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.168407 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.168384    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-tls-certs\") pod \"distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.169210 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.169190    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-tmp-dir\") pod \"distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.169757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.169736    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdgr9\" (UniqueName: \"kubernetes.io/projected/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-kube-api-access-kdgr9\") pod \"distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.170242 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.170222    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-home\") pod \"distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.170381 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.170365    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-dshm\") pod \"distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.170566 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.170538    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-home\") pod \"distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.170783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.168253    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-model-cache\") pod \"distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.170912 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.169636    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-tmp-dir\") pod \"distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.173524 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.173478    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-dshm\") pod \"distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.182064 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.182006    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-tls-certs\") pod \"distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.228484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.228419    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdgr9\" (UniqueName: \"kubernetes.io/projected/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-kube-api-access-kdgr9\") pod \"distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") " pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.310102 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.310071    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:03.771006 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:27:03.770974    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf4aa2ef_68c0_4d5d_b966_6c2f65766f7d.slice/crio-ca1b990be04299c6b8d7916bd36d3f0e646e4aeb344a97572964c8608232eb38 WatchSource:0}: Error finding container ca1b990be04299c6b8d7916bd36d3f0e646e4aeb344a97572964c8608232eb38: Status 404 returned error can't find the container with id ca1b990be04299c6b8d7916bd36d3f0e646e4aeb344a97572964c8608232eb38
Oct 30 23:27:03.771353 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:03.771331    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"]
Oct 30 23:27:04.214572 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:04.214529    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" event={"ID":"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d","Type":"ContainerStarted","Data":"9c756a6e3dae78ac950916b5c5bf3277065b49907ea1da17f42b4eb9e0c4919f"}
Oct 30 23:27:04.215077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:04.214579    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" event={"ID":"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d","Type":"ContainerStarted","Data":"ca1b990be04299c6b8d7916bd36d3f0e646e4aeb344a97572964c8608232eb38"}
Oct 30 23:27:04.310844 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:04.310812    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:10.522698 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:10.522638    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podStartSLOduration=8.522624308 podStartE2EDuration="8.522624308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:27:04.328498392 +0000 UTC m=+1210.216404747" watchObservedRunningTime="2026-10-30 23:27:10.522624308 +0000 UTC m=+1216.410530687"
Oct 30 23:27:10.524382 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:10.524363    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv"]
Oct 30 23:27:10.528293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:10.528278    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv"
Oct 30 23:27:10.585133 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:10.585101    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv"]
Oct 30 23:27:10.629897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:10.629869    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:27:10.630077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:10.629906    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:27:10.630077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:10.629906    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:27:10.630634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:10.630588    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:27:10.668643 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:10.668619    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3af4294-205a-4ec2-8114-df49ac3a1bbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b509852-l8zcv\" (UID: \"c3af4294-205a-4ec2-8114-df49ac3a1bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv"
Oct 30 23:27:10.668850 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:10.668831    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3af4294-205a-4ec2-8114-df49ac3a1bbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b509852-l8zcv\" (UID: \"c3af4294-205a-4ec2-8114-df49ac3a1bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv"
Oct 30 23:27:10.668961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:10.668947    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h7c9\" (UniqueName: \"kubernetes.io/projected/c3af4294-205a-4ec2-8114-df49ac3a1bbf-kube-api-access-5h7c9\") pod \"maas-api-revoke-sub-2b509852-l8zcv\" (UID: \"c3af4294-205a-4ec2-8114-df49ac3a1bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv"
Oct 30 23:27:10.770105 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:10.770066    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h7c9\" (UniqueName: \"kubernetes.io/projected/c3af4294-205a-4ec2-8114-df49ac3a1bbf-kube-api-access-5h7c9\") pod \"maas-api-revoke-sub-2b509852-l8zcv\" (UID: \"c3af4294-205a-4ec2-8114-df49ac3a1bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv"
Oct 30 23:27:10.770277 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:10.770121    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3af4294-205a-4ec2-8114-df49ac3a1bbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b509852-l8zcv\" (UID: \"c3af4294-205a-4ec2-8114-df49ac3a1bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv"
Oct 30 23:27:10.770277 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:10.770249    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3af4294-205a-4ec2-8114-df49ac3a1bbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b509852-l8zcv\" (UID: \"c3af4294-205a-4ec2-8114-df49ac3a1bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv"
Oct 30 23:27:10.770983 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:10.770905    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3af4294-205a-4ec2-8114-df49ac3a1bbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b509852-l8zcv\" (UID: \"c3af4294-205a-4ec2-8114-df49ac3a1bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv"
Oct 30 23:27:10.772843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:10.772792    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3af4294-205a-4ec2-8114-df49ac3a1bbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b509852-l8zcv\" (UID: \"c3af4294-205a-4ec2-8114-df49ac3a1bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv"
Oct 30 23:27:10.827029 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:10.827004    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h7c9\" (UniqueName: \"kubernetes.io/projected/c3af4294-205a-4ec2-8114-df49ac3a1bbf-kube-api-access-5h7c9\") pod \"maas-api-revoke-sub-2b509852-l8zcv\" (UID: \"c3af4294-205a-4ec2-8114-df49ac3a1bbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv"
Oct 30 23:27:10.841570 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:10.841551    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv"
Oct 30 23:27:11.214256 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:11.214228    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv"]
Oct 30 23:27:11.215077 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:27:11.215048    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3af4294_205a_4ec2_8114_df49ac3a1bbf.slice/crio-a8efeba3b51d86428d8528b52fceb193fc892dc7bdc9e26babe801bdba1d2347 WatchSource:0}: Error finding container a8efeba3b51d86428d8528b52fceb193fc892dc7bdc9e26babe801bdba1d2347: Status 404 returned error can't find the container with id a8efeba3b51d86428d8528b52fceb193fc892dc7bdc9e26babe801bdba1d2347
Oct 30 23:27:11.267044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:11.267011    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv" event={"ID":"c3af4294-205a-4ec2-8114-df49ac3a1bbf","Type":"ContainerStarted","Data":"a8efeba3b51d86428d8528b52fceb193fc892dc7bdc9e26babe801bdba1d2347"}
Oct 30 23:27:12.326625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:12.326572    5078 generic.go:356] "Generic (PLEG): container finished" podID="c3af4294-205a-4ec2-8114-df49ac3a1bbf" containerID="9ee54cedf4c019a27949e275c84c25512171fb3aee8b1a4a662c5f4bfe894167" exitCode=0
Oct 30 23:27:12.331229 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:12.331197    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv" event={"ID":"c3af4294-205a-4ec2-8114-df49ac3a1bbf","Type":"ContainerDied","Data":"9ee54cedf4c019a27949e275c84c25512171fb3aee8b1a4a662c5f4bfe894167"}
Oct 30 23:27:13.311202 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:13.311171    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:13.323964 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:13.323939    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:13.345681 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:13.345657    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:27:13.493657 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:13.493628    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv"
Oct 30 23:27:13.521208 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:13.521181    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3af4294-205a-4ec2-8114-df49ac3a1bbf-kube-api-access-5h7c9\" (UniqueName: \"kubernetes.io/projected/c3af4294-205a-4ec2-8114-df49ac3a1bbf-kube-api-access-5h7c9\") pod \"c3af4294-205a-4ec2-8114-df49ac3a1bbf\" (UID: \"c3af4294-205a-4ec2-8114-df49ac3a1bbf\") "
Oct 30 23:27:13.521405 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:13.521242    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3af4294-205a-4ec2-8114-df49ac3a1bbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3af4294-205a-4ec2-8114-df49ac3a1bbf-maas-api-openshift-ca\") pod \"c3af4294-205a-4ec2-8114-df49ac3a1bbf\" (UID: \"c3af4294-205a-4ec2-8114-df49ac3a1bbf\") "
Oct 30 23:27:13.521405 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:13.521346    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3af4294-205a-4ec2-8114-df49ac3a1bbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3af4294-205a-4ec2-8114-df49ac3a1bbf-maas-api-xks-ca\") pod \"c3af4294-205a-4ec2-8114-df49ac3a1bbf\" (UID: \"c3af4294-205a-4ec2-8114-df49ac3a1bbf\") "
Oct 30 23:27:13.521688 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:13.521661    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3af4294-205a-4ec2-8114-df49ac3a1bbf-maas-api-openshift-ca" pod "c3af4294-205a-4ec2-8114-df49ac3a1bbf" (UID: "c3af4294-205a-4ec2-8114-df49ac3a1bbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:27:13.521911 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:13.521889    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3af4294-205a-4ec2-8114-df49ac3a1bbf-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:27:13.523643 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:13.523612    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3af4294-205a-4ec2-8114-df49ac3a1bbf-maas-api-xks-ca" pod "c3af4294-205a-4ec2-8114-df49ac3a1bbf" (UID: "c3af4294-205a-4ec2-8114-df49ac3a1bbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:27:13.523733 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:13.523643    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3af4294-205a-4ec2-8114-df49ac3a1bbf-kube-api-access-5h7c9" pod "c3af4294-205a-4ec2-8114-df49ac3a1bbf" (UID: "c3af4294-205a-4ec2-8114-df49ac3a1bbf"). InnerVolumeSpecName "kube-api-access-5h7c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:27:13.622757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:13.622687    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3af4294-205a-4ec2-8114-df49ac3a1bbf-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:27:13.622757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:13.622713    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h7c9\" (UniqueName: \"kubernetes.io/projected/c3af4294-205a-4ec2-8114-df49ac3a1bbf-kube-api-access-5h7c9\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:27:14.341947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:14.341915    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv"
Oct 30 23:27:14.342101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:14.341952    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv" event={"ID":"c3af4294-205a-4ec2-8114-df49ac3a1bbf","Type":"ContainerDied","Data":"a8efeba3b51d86428d8528b52fceb193fc892dc7bdc9e26babe801bdba1d2347"}
Oct 30 23:27:14.342101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:14.342005    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8efeba3b51d86428d8528b52fceb193fc892dc7bdc9e26babe801bdba1d2347"
Oct 30 23:27:17.591065 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:27:17.591028    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:27:22.074537 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.074499    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb"]
Oct 30 23:27:22.117006 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.116972    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-12c3a3af-sffdb"]
Oct 30 23:27:22.242656 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.242624    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"]
Oct 30 23:27:22.243107 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.243095    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c3af4294-205a-4ec2-8114-df49ac3a1bbf" containerName="revoke-keys"
Oct 30 23:27:22.243166 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.243108    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3af4294-205a-4ec2-8114-df49ac3a1bbf" containerName="revoke-keys"
Oct 30 23:27:22.243219 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.243211    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3af4294-205a-4ec2-8114-df49ac3a1bbf" containerName="revoke-keys"
Oct 30 23:27:22.246776 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.246755    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.298174 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.298144    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f136b2e-6c4e-4ce3-a4ac-0900ba573099" path="/var/lib/kubelet/pods/8f136b2e-6c4e-4ce3-a4ac-0900ba573099/volumes"
Oct 30 23:27:22.300786 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.300766    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b81a7a\"/\"distinct-2-w1-b81a7a-kserve-self-signed-certs\""
Oct 30 23:27:22.300908 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.300801    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b81a7a\"/\"default-dockercfg-hwvw7\""
Oct 30 23:27:22.346731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.346653    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-dshm\") pod \"distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.346885 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.346749    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-tmp-dir\") pod \"distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.346885 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.346781    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-tls-certs\") pod \"distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.347009 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.346911    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-home\") pod \"distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.347104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.347086    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt45b\" (UniqueName: \"kubernetes.io/projected/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-kube-api-access-zt45b\") pod \"distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.347166 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.347125    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-model-cache\") pod \"distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.350135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.350116    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b81a7a\"/\"openshift-service-ca.crt\""
Oct 30 23:27:22.350358 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.350340    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b81a7a\"/\"kube-root-ca.crt\""
Oct 30 23:27:22.353727 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.353708    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"]
Oct 30 23:27:22.448508 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.448481    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt45b\" (UniqueName: \"kubernetes.io/projected/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-kube-api-access-zt45b\") pod \"distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.448508 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.448520    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-model-cache\") pod \"distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.448759 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.448579    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-dshm\") pod \"distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.448759 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.448656    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-tmp-dir\") pod \"distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.448759 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.448689    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-tls-certs\") pod \"distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.448898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.448763    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-home\") pod \"distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.449049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.449024    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-model-cache\") pod \"distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.449049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.449044    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-tmp-dir\") pod \"distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.449220 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.449108    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-home\") pod \"distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.451070 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.451051    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-dshm\") pod \"distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.451289 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.451272    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-tls-certs\") pod \"distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.506201 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.506171    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt45b\" (UniqueName: \"kubernetes.io/projected/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-kube-api-access-zt45b\") pod \"distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") " pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.559749 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.559722    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:22.939777 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:22.939745    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"]
Oct 30 23:27:22.939909 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:27:22.939775    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0be3c19_e6ea_49f2_b4c6_2945a49fcdd2.slice/crio-cd7ed027ade75ccef042943d27318eb34f5462e936b90736671689ab51c0fb7f WatchSource:0}: Error finding container cd7ed027ade75ccef042943d27318eb34f5462e936b90736671689ab51c0fb7f: Status 404 returned error can't find the container with id cd7ed027ade75ccef042943d27318eb34f5462e936b90736671689ab51c0fb7f
Oct 30 23:27:23.407460 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:23.407425    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" event={"ID":"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2","Type":"ContainerStarted","Data":"056d2f7a98b41e214d7be1c931ed645aeb8a0863ad1d8b10cea0d00b2929bbbe"}
Oct 30 23:27:23.407891 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:23.407472    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" event={"ID":"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2","Type":"ContainerStarted","Data":"cd7ed027ade75ccef042943d27318eb34f5462e936b90736671689ab51c0fb7f"}
Oct 30 23:27:23.539002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:23.538954    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podStartSLOduration=1.538939273 podStartE2EDuration="1.538939273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:27:23.537744837 +0000 UTC m=+1229.425651191" watchObservedRunningTime="2026-10-30 23:27:23.538939273 +0000 UTC m=+1229.426845625"
Oct 30 23:27:23.560363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:23.560337    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:27.009689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.009653    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5"]
Oct 30 23:27:27.013834 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.013813    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5"
Oct 30 23:27:27.066263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.066234    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:27:27.080480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.080443    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c573c188-ec6c-4465-98fd-edb374cd13f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d06b7cea-tclh5\" (UID: \"c573c188-ec6c-4465-98fd-edb374cd13f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5"
Oct 30 23:27:27.080653 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.080555    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5vrj\" (UniqueName: \"kubernetes.io/projected/c573c188-ec6c-4465-98fd-edb374cd13f7-kube-api-access-s5vrj\") pod \"maas-api-revoke-sub-d06b7cea-tclh5\" (UID: \"c573c188-ec6c-4465-98fd-edb374cd13f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5"
Oct 30 23:27:27.080743 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.080716    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c573c188-ec6c-4465-98fd-edb374cd13f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d06b7cea-tclh5\" (UID: \"c573c188-ec6c-4465-98fd-edb374cd13f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5"
Oct 30 23:27:27.115957 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.115924    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:27:27.115957 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.115927    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:27:27.116150 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.115977    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:27:27.119946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.119925    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5"]
Oct 30 23:27:27.181627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.181566    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c573c188-ec6c-4465-98fd-edb374cd13f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d06b7cea-tclh5\" (UID: \"c573c188-ec6c-4465-98fd-edb374cd13f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5"
Oct 30 23:27:27.181814 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.181793    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c573c188-ec6c-4465-98fd-edb374cd13f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d06b7cea-tclh5\" (UID: \"c573c188-ec6c-4465-98fd-edb374cd13f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5"
Oct 30 23:27:27.181901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.181886    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5vrj\" (UniqueName: \"kubernetes.io/projected/c573c188-ec6c-4465-98fd-edb374cd13f7-kube-api-access-s5vrj\") pod \"maas-api-revoke-sub-d06b7cea-tclh5\" (UID: \"c573c188-ec6c-4465-98fd-edb374cd13f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5"
Oct 30 23:27:27.182423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.182401    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c573c188-ec6c-4465-98fd-edb374cd13f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d06b7cea-tclh5\" (UID: \"c573c188-ec6c-4465-98fd-edb374cd13f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5"
Oct 30 23:27:27.184039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.184020    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c573c188-ec6c-4465-98fd-edb374cd13f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d06b7cea-tclh5\" (UID: \"c573c188-ec6c-4465-98fd-edb374cd13f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5"
Oct 30 23:27:27.239230 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.239196    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5vrj\" (UniqueName: \"kubernetes.io/projected/c573c188-ec6c-4465-98fd-edb374cd13f7-kube-api-access-s5vrj\") pod \"maas-api-revoke-sub-d06b7cea-tclh5\" (UID: \"c573c188-ec6c-4465-98fd-edb374cd13f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5"
Oct 30 23:27:27.329385 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.329344    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5"
Oct 30 23:27:27.704590 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.704563    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5"]
Oct 30 23:27:27.705007 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:27:27.704984    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc573c188_ec6c_4465_98fd_edb374cd13f7.slice/crio-f94028d5aff13cfec160fd229566b6a066a05fdd6fb76658eb5189887e83614f WatchSource:0}: Error finding container f94028d5aff13cfec160fd229566b6a066a05fdd6fb76658eb5189887e83614f: Status 404 returned error can't find the container with id f94028d5aff13cfec160fd229566b6a066a05fdd6fb76658eb5189887e83614f
Oct 30 23:27:27.758484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.758457    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"]
Oct 30 23:27:27.763778 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.763760    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:27.789200 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.789165    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-dshm\") pod \"distinct-2-w0-057303-kserve-578886f754-qrxhz\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") " pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:27.789339 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.789273    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07fffabb-6a6d-4ec5-9534-2c3af669a252-tls-certs\") pod \"distinct-2-w0-057303-kserve-578886f754-qrxhz\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") " pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:27.789339 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.789332    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-model-cache\") pod \"distinct-2-w0-057303-kserve-578886f754-qrxhz\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") " pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:27.789512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.789381    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-tmp-dir\") pod \"distinct-2-w0-057303-kserve-578886f754-qrxhz\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") " pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:27.789512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.789442    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-home\") pod \"distinct-2-w0-057303-kserve-578886f754-qrxhz\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") " pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:27.789512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.789501    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98ggf\" (UniqueName: \"kubernetes.io/projected/07fffabb-6a6d-4ec5-9534-2c3af669a252-kube-api-access-98ggf\") pod \"distinct-2-w0-057303-kserve-578886f754-qrxhz\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") " pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:27.820626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.817188    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-057303\"/\"distinct-2-w0-057303-kserve-self-signed-certs\""
Oct 30 23:27:27.828142 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.828112    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"]
Oct 30 23:27:27.890350 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.890320    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98ggf\" (UniqueName: \"kubernetes.io/projected/07fffabb-6a6d-4ec5-9534-2c3af669a252-kube-api-access-98ggf\") pod \"distinct-2-w0-057303-kserve-578886f754-qrxhz\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") " pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:27.890503 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.890416    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-dshm\") pod \"distinct-2-w0-057303-kserve-578886f754-qrxhz\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") " pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:27.890503 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.890489    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07fffabb-6a6d-4ec5-9534-2c3af669a252-tls-certs\") pod \"distinct-2-w0-057303-kserve-578886f754-qrxhz\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") " pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:27.890586 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.890558    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-model-cache\") pod \"distinct-2-w0-057303-kserve-578886f754-qrxhz\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") " pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:27.890675 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.890661    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-tmp-dir\") pod \"distinct-2-w0-057303-kserve-578886f754-qrxhz\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") " pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:27.890714 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.890702    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-home\") pod \"distinct-2-w0-057303-kserve-578886f754-qrxhz\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") " pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:27.890947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.890926    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-model-cache\") pod \"distinct-2-w0-057303-kserve-578886f754-qrxhz\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") " pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:27.891029 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.891003    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-tmp-dir\") pod \"distinct-2-w0-057303-kserve-578886f754-qrxhz\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") " pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:27.891101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.891052    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-home\") pod \"distinct-2-w0-057303-kserve-578886f754-qrxhz\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") " pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:27.966564 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.966444    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-dshm\") pod \"distinct-2-w0-057303-kserve-578886f754-qrxhz\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") " pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:27.966781 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.966755    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07fffabb-6a6d-4ec5-9534-2c3af669a252-tls-certs\") pod \"distinct-2-w0-057303-kserve-578886f754-qrxhz\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") " pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:27.966850 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:27.966785    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98ggf\" (UniqueName: \"kubernetes.io/projected/07fffabb-6a6d-4ec5-9534-2c3af669a252-kube-api-access-98ggf\") pod \"distinct-2-w0-057303-kserve-578886f754-qrxhz\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") " pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:28.075312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:28.075288    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:28.444267 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:28.444237    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5" event={"ID":"c573c188-ec6c-4465-98fd-edb374cd13f7","Type":"ContainerStarted","Data":"07df542f8d6055b8be34fe9e8ef102a552fd537ad5207c31dce8ef73757b2032"}
Oct 30 23:27:28.444445 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:28.444288    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5" event={"ID":"c573c188-ec6c-4465-98fd-edb374cd13f7","Type":"ContainerStarted","Data":"f94028d5aff13cfec160fd229566b6a066a05fdd6fb76658eb5189887e83614f"}
Oct 30 23:27:28.483667 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:28.483637    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"]
Oct 30 23:27:28.484342 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:27:28.484309    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07fffabb_6a6d_4ec5_9534_2c3af669a252.slice/crio-8ae1b2e3b92c75f9120adac53322ac0ec5741df72b5b1f2f5b8d6b70f1a640ad WatchSource:0}: Error finding container 8ae1b2e3b92c75f9120adac53322ac0ec5741df72b5b1f2f5b8d6b70f1a640ad: Status 404 returned error can't find the container with id 8ae1b2e3b92c75f9120adac53322ac0ec5741df72b5b1f2f5b8d6b70f1a640ad
Oct 30 23:27:28.561416 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:28.561367    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5" podStartSLOduration=2.561353307 podStartE2EDuration="2.561353307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:27:28.558477444 +0000 UTC m=+1234.446383789" watchObservedRunningTime="2026-10-30 23:27:28.561353307 +0000 UTC m=+1234.449259709"
Oct 30 23:27:29.452741 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:29.452701    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" event={"ID":"07fffabb-6a6d-4ec5-9534-2c3af669a252","Type":"ContainerStarted","Data":"c38856da170847c361c4e54e6af13d7806d17b2551b934eb28e37bbf650f9a6e"}
Oct 30 23:27:29.453171 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:29.452750    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" event={"ID":"07fffabb-6a6d-4ec5-9534-2c3af669a252","Type":"ContainerStarted","Data":"8ae1b2e3b92c75f9120adac53322ac0ec5741df72b5b1f2f5b8d6b70f1a640ad"}
Oct 30 23:27:29.454329 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:29.454305    5078 generic.go:356] "Generic (PLEG): container finished" podID="c573c188-ec6c-4465-98fd-edb374cd13f7" containerID="07df542f8d6055b8be34fe9e8ef102a552fd537ad5207c31dce8ef73757b2032" exitCode=0
Oct 30 23:27:29.454430 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:29.454370    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5" event={"ID":"c573c188-ec6c-4465-98fd-edb374cd13f7","Type":"ContainerDied","Data":"07df542f8d6055b8be34fe9e8ef102a552fd537ad5207c31dce8ef73757b2032"}
Oct 30 23:27:29.569884 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:29.569828    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podStartSLOduration=2.569815454 podStartE2EDuration="2.569815454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:27:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:27:29.567494982 +0000 UTC m=+1235.455401334" watchObservedRunningTime="2026-10-30 23:27:29.569815454 +0000 UTC m=+1235.457721807"
Oct 30 23:27:30.075682 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:30.075654    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:30.604453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:30.604429    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5"
Oct 30 23:27:30.641250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:30.641221    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c573c188-ec6c-4465-98fd-edb374cd13f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c573c188-ec6c-4465-98fd-edb374cd13f7-maas-api-xks-ca\") pod \"c573c188-ec6c-4465-98fd-edb374cd13f7\" (UID: \"c573c188-ec6c-4465-98fd-edb374cd13f7\") "
Oct 30 23:27:30.641376 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:30.641343    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c573c188-ec6c-4465-98fd-edb374cd13f7-kube-api-access-s5vrj\" (UniqueName: \"kubernetes.io/projected/c573c188-ec6c-4465-98fd-edb374cd13f7-kube-api-access-s5vrj\") pod \"c573c188-ec6c-4465-98fd-edb374cd13f7\" (UID: \"c573c188-ec6c-4465-98fd-edb374cd13f7\") "
Oct 30 23:27:30.641468 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:30.641456    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c573c188-ec6c-4465-98fd-edb374cd13f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c573c188-ec6c-4465-98fd-edb374cd13f7-maas-api-openshift-ca\") pod \"c573c188-ec6c-4465-98fd-edb374cd13f7\" (UID: \"c573c188-ec6c-4465-98fd-edb374cd13f7\") "
Oct 30 23:27:30.641862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:30.641835    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c573c188-ec6c-4465-98fd-edb374cd13f7-maas-api-openshift-ca" pod "c573c188-ec6c-4465-98fd-edb374cd13f7" (UID: "c573c188-ec6c-4465-98fd-edb374cd13f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:27:30.643484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:30.643458    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c573c188-ec6c-4465-98fd-edb374cd13f7-maas-api-xks-ca" pod "c573c188-ec6c-4465-98fd-edb374cd13f7" (UID: "c573c188-ec6c-4465-98fd-edb374cd13f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:27:30.643803 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:30.643786    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c573c188-ec6c-4465-98fd-edb374cd13f7-kube-api-access-s5vrj" pod "c573c188-ec6c-4465-98fd-edb374cd13f7" (UID: "c573c188-ec6c-4465-98fd-edb374cd13f7"). InnerVolumeSpecName "kube-api-access-s5vrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:27:30.743318 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:30.743245    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c573c188-ec6c-4465-98fd-edb374cd13f7-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:27:30.743318 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:30.743273    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5vrj\" (UniqueName: \"kubernetes.io/projected/c573c188-ec6c-4465-98fd-edb374cd13f7-kube-api-access-s5vrj\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:27:30.743318 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:30.743286    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c573c188-ec6c-4465-98fd-edb374cd13f7-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:27:31.468720 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:31.468695    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5"
Oct 30 23:27:31.468930 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:31.468738    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5" event={"ID":"c573c188-ec6c-4465-98fd-edb374cd13f7","Type":"ContainerDied","Data":"f94028d5aff13cfec160fd229566b6a066a05fdd6fb76658eb5189887e83614f"}
Oct 30 23:27:31.468930 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:31.468797    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f94028d5aff13cfec160fd229566b6a066a05fdd6fb76658eb5189887e83614f"
Oct 30 23:27:32.560689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:32.560657    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:32.573126 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:32.573101    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:33.047844 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:33.047814    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8"]
Oct 30 23:27:33.051129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:33.051107    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b360db24-b5dz8"]
Oct 30 23:27:33.492836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:33.492758    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:27:34.298855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:34.298817    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db094b09-fcd8-46dc-96f1-674cdd130756" path="/var/lib/kubelet/pods/db094b09-fcd8-46dc-96f1-674cdd130756/volumes"
Oct 30 23:27:37.789230 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:37.789201    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw"]
Oct 30 23:27:37.789770 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:37.789756    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c573c188-ec6c-4465-98fd-edb374cd13f7" containerName="revoke-keys"
Oct 30 23:27:37.789824 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:37.789771    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c573c188-ec6c-4465-98fd-edb374cd13f7" containerName="revoke-keys"
Oct 30 23:27:37.789873 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:37.789864    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c573c188-ec6c-4465-98fd-edb374cd13f7" containerName="revoke-keys"
Oct 30 23:27:37.794221 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:37.794205    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw"
Oct 30 23:27:37.849271 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:37.849238    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw"]
Oct 30 23:27:37.894083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:37.894048    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:27:37.894288 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:37.894095    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:27:37.894288 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:37.894181    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:27:37.895823 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:37.895807    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:27:37.933955 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:37.933920    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b479e9-979a-4251-9278-f478dec4f82f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bb1a5d5-xtscw\" (UID: \"f0b479e9-979a-4251-9278-f478dec4f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw"
Oct 30 23:27:37.934169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:37.934152    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4g9z\" (UniqueName: \"kubernetes.io/projected/f0b479e9-979a-4251-9278-f478dec4f82f-kube-api-access-d4g9z\") pod \"maas-api-revoke-sub-9bb1a5d5-xtscw\" (UID: \"f0b479e9-979a-4251-9278-f478dec4f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw"
Oct 30 23:27:37.934232 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:37.934217    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b479e9-979a-4251-9278-f478dec4f82f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bb1a5d5-xtscw\" (UID: \"f0b479e9-979a-4251-9278-f478dec4f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw"
Oct 30 23:27:38.035479 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:38.035447    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b479e9-979a-4251-9278-f478dec4f82f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bb1a5d5-xtscw\" (UID: \"f0b479e9-979a-4251-9278-f478dec4f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw"
Oct 30 23:27:38.035670 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:38.035491    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b479e9-979a-4251-9278-f478dec4f82f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bb1a5d5-xtscw\" (UID: \"f0b479e9-979a-4251-9278-f478dec4f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw"
Oct 30 23:27:38.035722 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:38.035686    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4g9z\" (UniqueName: \"kubernetes.io/projected/f0b479e9-979a-4251-9278-f478dec4f82f-kube-api-access-d4g9z\") pod \"maas-api-revoke-sub-9bb1a5d5-xtscw\" (UID: \"f0b479e9-979a-4251-9278-f478dec4f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw"
Oct 30 23:27:38.036306 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:38.036237    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b479e9-979a-4251-9278-f478dec4f82f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9bb1a5d5-xtscw\" (UID: \"f0b479e9-979a-4251-9278-f478dec4f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw"
Oct 30 23:27:38.038004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:38.037980    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b479e9-979a-4251-9278-f478dec4f82f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9bb1a5d5-xtscw\" (UID: \"f0b479e9-979a-4251-9278-f478dec4f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw"
Oct 30 23:27:38.075440 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:38.075412    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:38.088230 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:38.088207    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:38.092947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:38.092918    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4g9z\" (UniqueName: \"kubernetes.io/projected/f0b479e9-979a-4251-9278-f478dec4f82f-kube-api-access-d4g9z\") pod \"maas-api-revoke-sub-9bb1a5d5-xtscw\" (UID: \"f0b479e9-979a-4251-9278-f478dec4f82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw"
Oct 30 23:27:38.107504 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:38.107481    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw"
Oct 30 23:27:38.483731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:38.483707    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw"]
Oct 30 23:27:38.485030 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:27:38.485005    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0b479e9_979a_4251_9278_f478dec4f82f.slice/crio-eb9caf3ed373a3ec49f7fa9658263aeff20dc34b8ba930b67844c75090637f75 WatchSource:0}: Error finding container eb9caf3ed373a3ec49f7fa9658263aeff20dc34b8ba930b67844c75090637f75: Status 404 returned error can't find the container with id eb9caf3ed373a3ec49f7fa9658263aeff20dc34b8ba930b67844c75090637f75
Oct 30 23:27:38.515135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:38.515103    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw" event={"ID":"f0b479e9-979a-4251-9278-f478dec4f82f","Type":"ContainerStarted","Data":"eb9caf3ed373a3ec49f7fa9658263aeff20dc34b8ba930b67844c75090637f75"}
Oct 30 23:27:38.527067 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:38.527045    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:27:39.523556 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:39.523519    5078 generic.go:356] "Generic (PLEG): container finished" podID="f0b479e9-979a-4251-9278-f478dec4f82f" containerID="e628021035c74b5ab1dc6f0c8a0fc0d49450786f94ba144defb1c4ef1edcddbb" exitCode=0
Oct 30 23:27:39.523969 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:39.523616    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw" event={"ID":"f0b479e9-979a-4251-9278-f478dec4f82f","Type":"ContainerDied","Data":"e628021035c74b5ab1dc6f0c8a0fc0d49450786f94ba144defb1c4ef1edcddbb"}
Oct 30 23:27:40.687455 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:40.687430    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw"
Oct 30 23:27:40.771502 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:40.771470    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0b479e9-979a-4251-9278-f478dec4f82f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b479e9-979a-4251-9278-f478dec4f82f-maas-api-openshift-ca\") pod \"f0b479e9-979a-4251-9278-f478dec4f82f\" (UID: \"f0b479e9-979a-4251-9278-f478dec4f82f\") "
Oct 30 23:27:40.771706 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:40.771561    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0b479e9-979a-4251-9278-f478dec4f82f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b479e9-979a-4251-9278-f478dec4f82f-maas-api-xks-ca\") pod \"f0b479e9-979a-4251-9278-f478dec4f82f\" (UID: \"f0b479e9-979a-4251-9278-f478dec4f82f\") "
Oct 30 23:27:40.771706 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:40.771688    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0b479e9-979a-4251-9278-f478dec4f82f-kube-api-access-d4g9z\" (UniqueName: \"kubernetes.io/projected/f0b479e9-979a-4251-9278-f478dec4f82f-kube-api-access-d4g9z\") pod \"f0b479e9-979a-4251-9278-f478dec4f82f\" (UID: \"f0b479e9-979a-4251-9278-f478dec4f82f\") "
Oct 30 23:27:40.771924 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:40.771892    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0b479e9-979a-4251-9278-f478dec4f82f-maas-api-openshift-ca" pod "f0b479e9-979a-4251-9278-f478dec4f82f" (UID: "f0b479e9-979a-4251-9278-f478dec4f82f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:27:40.772217 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:40.772198    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f0b479e9-979a-4251-9278-f478dec4f82f-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:27:40.773924 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:40.773899    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0b479e9-979a-4251-9278-f478dec4f82f-kube-api-access-d4g9z" pod "f0b479e9-979a-4251-9278-f478dec4f82f" (UID: "f0b479e9-979a-4251-9278-f478dec4f82f"). InnerVolumeSpecName "kube-api-access-d4g9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:27:40.773924 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:40.773911    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0b479e9-979a-4251-9278-f478dec4f82f-maas-api-xks-ca" pod "f0b479e9-979a-4251-9278-f478dec4f82f" (UID: "f0b479e9-979a-4251-9278-f478dec4f82f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:27:40.877623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:40.874635    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f0b479e9-979a-4251-9278-f478dec4f82f-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:27:40.877623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:40.874672    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4g9z\" (UniqueName: \"kubernetes.io/projected/f0b479e9-979a-4251-9278-f478dec4f82f-kube-api-access-d4g9z\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:27:41.547136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:41.547081    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw" event={"ID":"f0b479e9-979a-4251-9278-f478dec4f82f","Type":"ContainerDied","Data":"eb9caf3ed373a3ec49f7fa9658263aeff20dc34b8ba930b67844c75090637f75"}
Oct 30 23:27:41.547519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:41.547498    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb9caf3ed373a3ec49f7fa9658263aeff20dc34b8ba930b67844c75090637f75"
Oct 30 23:27:41.547688 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:41.547443    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw"
Oct 30 23:27:43.245273 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.245233    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"]
Oct 30 23:27:43.245854 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.245839    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f0b479e9-979a-4251-9278-f478dec4f82f" containerName="revoke-keys"
Oct 30 23:27:43.245910 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.245855    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0b479e9-979a-4251-9278-f478dec4f82f" containerName="revoke-keys"
Oct 30 23:27:43.245967 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.245958    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0b479e9-979a-4251-9278-f478dec4f82f" containerName="revoke-keys"
Oct 30 23:27:43.249551 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.249533    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.349340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.349312    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-istio-data\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.349503 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.349363    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660a691a-733b-4ea7-a3db-4c55f649b099-istiod-ca-cert\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.349503 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.349391    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660a691a-733b-4ea7-a3db-4c55f649b099-istio-token\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.349503 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.349487    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-credential-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.349689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.349572    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-istio-envoy\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.349689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.349668    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gtlx\" (UniqueName: \"kubernetes.io/projected/660a691a-733b-4ea7-a3db-4c55f649b099-kube-api-access-2gtlx\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.349800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.349695    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-workload-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.349856 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.349813    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-workload-certs\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.350002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.349979    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660a691a-733b-4ea7-a3db-4c55f649b099-istio-podinfo\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.357235 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.357207    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"]
Oct 30 23:27:43.451663 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.451634    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gtlx\" (UniqueName: \"kubernetes.io/projected/660a691a-733b-4ea7-a3db-4c55f649b099-kube-api-access-2gtlx\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.451852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.451672    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-workload-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.451852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.451724    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-workload-certs\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.451852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.451813    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660a691a-733b-4ea7-a3db-4c55f649b099-istio-podinfo\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.451852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.451836    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-istio-data\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.452076 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.451872    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660a691a-733b-4ea7-a3db-4c55f649b099-istiod-ca-cert\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.452076 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.451900    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660a691a-733b-4ea7-a3db-4c55f649b099-istio-token\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.452076 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.451942    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-credential-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.452076 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.451973    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-istio-envoy\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.452076 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.452059    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-workload-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.452297 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.452143    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-workload-certs\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.452297 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.452204    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-istio-data\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.452363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.452327    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-credential-socket\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.452705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.452545    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660a691a-733b-4ea7-a3db-4c55f649b099-istiod-ca-cert\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.454260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.454240    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-istio-envoy\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.454475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.454450    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660a691a-733b-4ea7-a3db-4c55f649b099-istio-podinfo\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.509020 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.508960    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660a691a-733b-4ea7-a3db-4c55f649b099-istio-token\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.509342 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.509316    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gtlx\" (UniqueName: \"kubernetes.io/projected/660a691a-733b-4ea7-a3db-4c55f649b099-kube-api-access-2gtlx\") pod \"e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") " pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.564339 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.564310    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:43.911272 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.911243    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"]
Oct 30 23:27:43.916367 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:43.916352    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.010684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.009250    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"]
Oct 30 23:27:44.013964 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.013940    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"]
Oct 30 23:27:44.016105 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.016067    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:27:44.016196 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.016153    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:27:44.016243 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.016202    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:27:44.059371 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.059329    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.059519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.059386    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77jcx\" (UniqueName: \"kubernetes.io/projected/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-kube-api-access-77jcx\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.059519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.059502    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.059660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.059648    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.059731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.059679    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.059853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.059827    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.059924 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.059880    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.060064 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.060047    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.060180 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.060158    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.162027 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.161935    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.162027 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.161997    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.162266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.162035    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.162266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.162055    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77jcx\" (UniqueName: \"kubernetes.io/projected/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-kube-api-access-77jcx\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.162266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.162090    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.162266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.162163    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.162266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.162196    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.162266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.162244    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.162582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.162268    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.162661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.162622    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.162713 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.162673    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.162713 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.162699    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.162844 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.162821    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.162959 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.162939    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.164770 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.164749    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.164940 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.164924    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.220637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.220575    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.220773 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.220669    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77jcx\" (UniqueName: \"kubernetes.io/projected/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-kube-api-access-77jcx\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.230757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.230731    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:44.612796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.612769    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"]
Oct 30 23:27:44.613773 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:27:44.613739    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3adadcee_68a6_4a1c_9c0c_d2fa9106ef97.slice/crio-3d437f6a9ebc8f9976de8595a682b4789c93482200f9674a06c4ba2a79b1f328 WatchSource:0}: Error finding container 3d437f6a9ebc8f9976de8595a682b4789c93482200f9674a06c4ba2a79b1f328: Status 404 returned error can't find the container with id 3d437f6a9ebc8f9976de8595a682b4789c93482200f9674a06c4ba2a79b1f328
Oct 30 23:27:44.616684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.616651    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:27:44.616783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.616732    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:27:44.616841 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.616778    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:27:44.650733 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.650700    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j" event={"ID":"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97","Type":"ContainerStarted","Data":"3d437f6a9ebc8f9976de8595a682b4789c93482200f9674a06c4ba2a79b1f328"}
Oct 30 23:27:44.652344 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.652325    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj" event={"ID":"660a691a-733b-4ea7-a3db-4c55f649b099","Type":"ContainerStarted","Data":"b1346fa110f48ab56388373552c87d5c80fef4be79cac212bf5c6e86a09148a7"}
Oct 30 23:27:44.652432 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.652362    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj" event={"ID":"660a691a-733b-4ea7-a3db-4c55f649b099","Type":"ContainerStarted","Data":"90def38b0b85eb0709f493408fb11e7bc33644d3438694ce1b181d6c0176808e"}
Oct 30 23:27:44.772630 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:44.772345    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj" podStartSLOduration=1.7723263 podStartE2EDuration="1.7723263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:27:44.771060796 +0000 UTC m=+1250.658967148" watchObservedRunningTime="2026-10-30 23:27:44.7723263 +0000 UTC m=+1250.660232655"
Oct 30 23:27:45.569624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:45.565956    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:45.592623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:45.585735    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 23:27:45.592623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:45.585799    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj" podUID="660a691a-733b-4ea7-a3db-4c55f649b099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 23:27:45.697622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:45.696890    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j" event={"ID":"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97","Type":"ContainerStarted","Data":"d039eeba52ec6ba8dce5bbdcb7e5dfc16bd0fd4f72c7d368fde0c8611de62583"}
Oct 30 23:27:45.939174 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:45.939077    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j" podStartSLOduration=2.939062902 podStartE2EDuration="2.939062902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:27:45.887148268 +0000 UTC m=+1251.775054624" watchObservedRunningTime="2026-10-30 23:27:45.939062902 +0000 UTC m=+1251.826969255"
Oct 30 23:27:46.231188 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:46.231105    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:46.232583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:46.232564    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 23:27:46.232661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:46.232627    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j" podUID="3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 23:27:46.565622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:46.565013    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 23:27:46.565622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:46.565088    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj" podUID="660a691a-733b-4ea7-a3db-4c55f649b099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 23:27:47.232935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:47.231581    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 23:27:47.232935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:47.231664    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j" podUID="3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 23:27:47.567625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:47.565948    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 23:27:47.567625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:47.566009    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj" podUID="660a691a-733b-4ea7-a3db-4c55f649b099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 23:27:47.633161 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:27:47.633127    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:27:48.231536 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:48.231503    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 23:27:48.231823 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:48.231794    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j" podUID="3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 23:27:48.565067 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:48.565026    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 23:27:48.565700 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:48.565666    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj" podUID="660a691a-733b-4ea7-a3db-4c55f649b099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 23:27:49.048436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:49.048404    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp"]
Oct 30 23:27:49.108267 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:49.108228    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae3a479f-7l8fp"]
Oct 30 23:27:49.231401 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:49.231371    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 23:27:49.231569 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:49.231419    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j" podUID="3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 23:27:49.576623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:49.575061    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 23:27:49.576623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:49.575124    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj" podUID="660a691a-733b-4ea7-a3db-4c55f649b099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 23:27:50.231805 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:50.231769    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 23:27:50.232152 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:50.231830    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j" podUID="3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 23:27:50.300245 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:50.300210    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad06fb97-40ea-4d04-9f52-89c8f2ec85db" path="/var/lib/kubelet/pods/ad06fb97-40ea-4d04-9f52-89c8f2ec85db/volumes"
Oct 30 23:27:50.568619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:50.567891    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused" start-of-body=
Oct 30 23:27:50.568619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:50.567947    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj" podUID="660a691a-733b-4ea7-a3db-4c55f649b099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": dial tcp 10.133.0.86:15021: connect: connection refused"
Oct 30 23:27:51.231226 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.231191    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 23:27:51.231613 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.231261    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j" podUID="3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 23:27:51.257428 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.257398    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw"]
Oct 30 23:27:51.263788 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.263767    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw"
Oct 30 23:27:51.317752 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.317714    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw"]
Oct 30 23:27:51.335827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.335794    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:27:51.349638 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.349608    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52318041-8b2f-4a12-9ba4-da9e1a0bb135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d523e83-m4ljw\" (UID: \"52318041-8b2f-4a12-9ba4-da9e1a0bb135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw"
Oct 30 23:27:51.349805 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.349757    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kb24h\" (UniqueName: \"kubernetes.io/projected/52318041-8b2f-4a12-9ba4-da9e1a0bb135-kube-api-access-kb24h\") pod \"maas-api-revoke-sub-2d523e83-m4ljw\" (UID: \"52318041-8b2f-4a12-9ba4-da9e1a0bb135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw"
Oct 30 23:27:51.349894 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.349879    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52318041-8b2f-4a12-9ba4-da9e1a0bb135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d523e83-m4ljw\" (UID: \"52318041-8b2f-4a12-9ba4-da9e1a0bb135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw"
Oct 30 23:27:51.362102 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.362068    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:27:51.362354 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.362322    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:27:51.362717 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.362699    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:27:51.451191 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.451161    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52318041-8b2f-4a12-9ba4-da9e1a0bb135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d523e83-m4ljw\" (UID: \"52318041-8b2f-4a12-9ba4-da9e1a0bb135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw"
Oct 30 23:27:51.451396 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.451239    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52318041-8b2f-4a12-9ba4-da9e1a0bb135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d523e83-m4ljw\" (UID: \"52318041-8b2f-4a12-9ba4-da9e1a0bb135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw"
Oct 30 23:27:51.451396 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.451356    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kb24h\" (UniqueName: \"kubernetes.io/projected/52318041-8b2f-4a12-9ba4-da9e1a0bb135-kube-api-access-kb24h\") pod \"maas-api-revoke-sub-2d523e83-m4ljw\" (UID: \"52318041-8b2f-4a12-9ba4-da9e1a0bb135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw"
Oct 30 23:27:51.452070 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.452046    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52318041-8b2f-4a12-9ba4-da9e1a0bb135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d523e83-m4ljw\" (UID: \"52318041-8b2f-4a12-9ba4-da9e1a0bb135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw"
Oct 30 23:27:51.454017 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.453987    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52318041-8b2f-4a12-9ba4-da9e1a0bb135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d523e83-m4ljw\" (UID: \"52318041-8b2f-4a12-9ba4-da9e1a0bb135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw"
Oct 30 23:27:51.509084 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.509012    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kb24h\" (UniqueName: \"kubernetes.io/projected/52318041-8b2f-4a12-9ba4-da9e1a0bb135-kube-api-access-kb24h\") pod \"maas-api-revoke-sub-2d523e83-m4ljw\" (UID: \"52318041-8b2f-4a12-9ba4-da9e1a0bb135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw"
Oct 30 23:27:51.580471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.580438    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw"
Oct 30 23:27:51.621260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.621228    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:51.622939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.622775    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:51.623717 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:51.623696    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:27:52.021767 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:52.021730    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw"]
Oct 30 23:27:52.066318 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:27:52.065682    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52318041_8b2f_4a12_9ba4_da9e1a0bb135.slice/crio-29bc00b77a72c66b3768cc29310e3c09c2877108f353629ae1bec7d40f1f608a WatchSource:0}: Error finding container 29bc00b77a72c66b3768cc29310e3c09c2877108f353629ae1bec7d40f1f608a: Status 404 returned error can't find the container with id 29bc00b77a72c66b3768cc29310e3c09c2877108f353629ae1bec7d40f1f608a
Oct 30 23:27:52.344066 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:52.344033    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:52.344589 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:52.344450    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:52.345635 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:52.345584    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:27:52.793980 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:52.793930    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw" event={"ID":"52318041-8b2f-4a12-9ba4-da9e1a0bb135","Type":"ContainerStarted","Data":"2dec40a2a0b46f93bd3f217b176f452572378844593884fda70b9e9fb523f1af"}
Oct 30 23:27:52.794170 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:52.794004    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw" event={"ID":"52318041-8b2f-4a12-9ba4-da9e1a0bb135","Type":"ContainerStarted","Data":"29bc00b77a72c66b3768cc29310e3c09c2877108f353629ae1bec7d40f1f608a"}
Oct 30 23:27:52.907389 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:52.907334    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw" podStartSLOduration=1.907321429 podStartE2EDuration="1.907321429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:27:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:27:52.905945064 +0000 UTC m=+1258.793851418" watchObservedRunningTime="2026-10-30 23:27:52.907321429 +0000 UTC m=+1258.795227781"
Oct 30 23:27:53.803236 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:53.803201    5078 generic.go:356] "Generic (PLEG): container finished" podID="52318041-8b2f-4a12-9ba4-da9e1a0bb135" containerID="2dec40a2a0b46f93bd3f217b176f452572378844593884fda70b9e9fb523f1af" exitCode=0
Oct 30 23:27:53.803661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:53.803294    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw" event={"ID":"52318041-8b2f-4a12-9ba4-da9e1a0bb135","Type":"ContainerDied","Data":"2dec40a2a0b46f93bd3f217b176f452572378844593884fda70b9e9fb523f1af"}
Oct 30 23:27:54.964281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:54.964257    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw"
Oct 30 23:27:54.986464 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:54.986441    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52318041-8b2f-4a12-9ba4-da9e1a0bb135-kube-api-access-kb24h\" (UniqueName: \"kubernetes.io/projected/52318041-8b2f-4a12-9ba4-da9e1a0bb135-kube-api-access-kb24h\") pod \"52318041-8b2f-4a12-9ba4-da9e1a0bb135\" (UID: \"52318041-8b2f-4a12-9ba4-da9e1a0bb135\") "
Oct 30 23:27:54.986739 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:54.986720    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52318041-8b2f-4a12-9ba4-da9e1a0bb135-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52318041-8b2f-4a12-9ba4-da9e1a0bb135-maas-api-openshift-ca\") pod \"52318041-8b2f-4a12-9ba4-da9e1a0bb135\" (UID: \"52318041-8b2f-4a12-9ba4-da9e1a0bb135\") "
Oct 30 23:27:54.986823 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:54.986763    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52318041-8b2f-4a12-9ba4-da9e1a0bb135-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52318041-8b2f-4a12-9ba4-da9e1a0bb135-maas-api-xks-ca\") pod \"52318041-8b2f-4a12-9ba4-da9e1a0bb135\" (UID: \"52318041-8b2f-4a12-9ba4-da9e1a0bb135\") "
Oct 30 23:27:54.987098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:54.987075    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52318041-8b2f-4a12-9ba4-da9e1a0bb135-maas-api-openshift-ca" pod "52318041-8b2f-4a12-9ba4-da9e1a0bb135" (UID: "52318041-8b2f-4a12-9ba4-da9e1a0bb135"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:27:54.987236 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:54.987223    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52318041-8b2f-4a12-9ba4-da9e1a0bb135-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:27:54.988609 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:54.988572    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52318041-8b2f-4a12-9ba4-da9e1a0bb135-kube-api-access-kb24h" pod "52318041-8b2f-4a12-9ba4-da9e1a0bb135" (UID: "52318041-8b2f-4a12-9ba4-da9e1a0bb135"). InnerVolumeSpecName "kube-api-access-kb24h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:27:54.988823 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:54.988802    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52318041-8b2f-4a12-9ba4-da9e1a0bb135-maas-api-xks-ca" pod "52318041-8b2f-4a12-9ba4-da9e1a0bb135" (UID: "52318041-8b2f-4a12-9ba4-da9e1a0bb135"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:27:55.088525 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:55.088456    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52318041-8b2f-4a12-9ba4-da9e1a0bb135-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:27:55.088525 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:55.088492    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kb24h\" (UniqueName: \"kubernetes.io/projected/52318041-8b2f-4a12-9ba4-da9e1a0bb135-kube-api-access-kb24h\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:27:55.818316 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:55.818286    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw"
Oct 30 23:27:55.818501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:55.818324    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw" event={"ID":"52318041-8b2f-4a12-9ba4-da9e1a0bb135","Type":"ContainerDied","Data":"29bc00b77a72c66b3768cc29310e3c09c2877108f353629ae1bec7d40f1f608a"}
Oct 30 23:27:55.818501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:55.818375    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29bc00b77a72c66b3768cc29310e3c09c2877108f353629ae1bec7d40f1f608a"
Oct 30 23:27:59.013831 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.013790    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"]
Oct 30 23:27:59.014405 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.014389    5078 cpu_manager.go:414] "containerMap: removing container" podUID="52318041-8b2f-4a12-9ba4-da9e1a0bb135" containerName="revoke-keys"
Oct 30 23:27:59.014449 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.014408    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52318041-8b2f-4a12-9ba4-da9e1a0bb135" containerName="revoke-keys"
Oct 30 23:27:59.014528 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.014519    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="52318041-8b2f-4a12-9ba4-da9e1a0bb135" containerName="revoke-keys"
Oct 30 23:27:59.018144 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.018124    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.076931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.076899    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"]
Oct 30 23:27:59.140270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.140230    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.140270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.140285    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.140486 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.140433    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.140521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.140495    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.140584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.140567    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.140711 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.140698    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b964ddd5-6455-412c-85a8-07cd6b15cf40-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.140758 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.140743    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4pxv\" (UniqueName: \"kubernetes.io/projected/b964ddd5-6455-412c-85a8-07cd6b15cf40-kube-api-access-c4pxv\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.140832 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.140813    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.140895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.140884    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.241698 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.241665    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b964ddd5-6455-412c-85a8-07cd6b15cf40-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.241887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.241738    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4pxv\" (UniqueName: \"kubernetes.io/projected/b964ddd5-6455-412c-85a8-07cd6b15cf40-kube-api-access-c4pxv\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.241887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.241762    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.241887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.241788    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.241887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.241817    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.241887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.241841    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.242181 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.241931    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.242181 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.241986    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.242181 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.242024    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.242329 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.242226    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.242329 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.242253    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.242329 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.242320    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.242476 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.242440    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b964ddd5-6455-412c-85a8-07cd6b15cf40-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.242590 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.242566    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.244545 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.244522    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.244669 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.244576    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.299489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.299460    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.299660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.299563    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4pxv\" (UniqueName: \"kubernetes.io/projected/b964ddd5-6455-412c-85a8-07cd6b15cf40-kube-api-access-c4pxv\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.331424 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.331400    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:27:59.758041 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.758014    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"]
Oct 30 23:27:59.758863 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:27:59.758836    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb964ddd5_6455_412c_85a8_07cd6b15cf40.slice/crio-802fb72cca19d21574712615206444a18eb23bdb3f87bd8ec44bf2f89740a8f2 WatchSource:0}: Error finding container 802fb72cca19d21574712615206444a18eb23bdb3f87bd8ec44bf2f89740a8f2: Status 404 returned error can't find the container with id 802fb72cca19d21574712615206444a18eb23bdb3f87bd8ec44bf2f89740a8f2
Oct 30 23:27:59.761955 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.761916    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:27:59.762041 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.762001    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:27:59.762081 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.762040    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:27:59.847491 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.847465    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc" event={"ID":"b964ddd5-6455-412c-85a8-07cd6b15cf40","Type":"ContainerStarted","Data":"15609963fa4bc950ed64278e63d07beadd3fe5117ee162c78909feb234f4a781"}
Oct 30 23:27:59.847639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.847527    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc" event={"ID":"b964ddd5-6455-412c-85a8-07cd6b15cf40","Type":"ContainerStarted","Data":"802fb72cca19d21574712615206444a18eb23bdb3f87bd8ec44bf2f89740a8f2"}
Oct 30 23:27:59.966075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:27:59.966027    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc" podStartSLOduration=1.9660139079999999 podStartE2EDuration="1.966013908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:27:59.964838239 +0000 UTC m=+1265.852744593" watchObservedRunningTime="2026-10-30 23:27:59.966013908 +0000 UTC m=+1265.853920260"
Oct 30 23:28:00.331748 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:00.331719    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:28:00.332955 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:00.332931    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:28:00.333031 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:00.332982    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc" podUID="b964ddd5-6455-412c-85a8-07cd6b15cf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:28:01.332477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:01.332439    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:28:01.332922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:01.332494    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc" podUID="b964ddd5-6455-412c-85a8-07cd6b15cf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:28:02.332292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:02.332261    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:28:02.332470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:02.332327    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc" podUID="b964ddd5-6455-412c-85a8-07cd6b15cf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:28:03.332589 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:03.332507    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:28:03.332589 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:03.332559    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc" podUID="b964ddd5-6455-412c-85a8-07cd6b15cf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:28:04.332170 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:04.332135    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:28:04.332353 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:04.332184    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc" podUID="b964ddd5-6455-412c-85a8-07cd6b15cf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:28:05.332721 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:05.332690    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:28:05.333193 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:05.332741    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc" podUID="b964ddd5-6455-412c-85a8-07cd6b15cf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:28:06.332070 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:06.332034    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 23:28:06.332251 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:06.332098    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc" podUID="b964ddd5-6455-412c-85a8-07cd6b15cf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 23:28:07.336748 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:07.336720    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:28:07.337195 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:07.337176    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:28:07.337880 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:07.337861    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:28:07.934538 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:07.934497    5078 generic.go:356] "Generic (PLEG): container finished" podID="5465b09b-ab0c-462e-9e44-c3f884253bb6" containerID="19a4ea1a3c4d7c6ed1afd181c6dda344a29c3310a4e4fe15363f1c95b922945d" exitCode=0
Oct 30 23:28:07.934758 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:07.934574    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fbxsv-m7vqv" event={"ID":"5465b09b-ab0c-462e-9e44-c3f884253bb6","Type":"ContainerDied","Data":"19a4ea1a3c4d7c6ed1afd181c6dda344a29c3310a4e4fe15363f1c95b922945d"}
Oct 30 23:28:08.715330 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:08.715297    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql"]
Oct 30 23:28:08.719433 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:08.719417    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql"
Oct 30 23:28:08.775733 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:08.775707    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql"]
Oct 30 23:28:08.810663 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:08.810631    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fabb604-c8a4-4e4e-a1ee-690b494fda35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34ef737b-fktql\" (UID: \"0fabb604-c8a4-4e4e-a1ee-690b494fda35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql"
Oct 30 23:28:08.810830 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:08.810687    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fabb604-c8a4-4e4e-a1ee-690b494fda35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34ef737b-fktql\" (UID: \"0fabb604-c8a4-4e4e-a1ee-690b494fda35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql"
Oct 30 23:28:08.810896 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:08.810846    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmrcf\" (UniqueName: \"kubernetes.io/projected/0fabb604-c8a4-4e4e-a1ee-690b494fda35-kube-api-access-lmrcf\") pod \"maas-api-revoke-sub-34ef737b-fktql\" (UID: \"0fabb604-c8a4-4e4e-a1ee-690b494fda35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql"
Oct 30 23:28:08.820719 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:08.820697    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:28:08.820863 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:08.820850    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:28:08.820913 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:08.820869    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:28:08.821115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:08.821099    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:28:08.912312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:08.912286    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fabb604-c8a4-4e4e-a1ee-690b494fda35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34ef737b-fktql\" (UID: \"0fabb604-c8a4-4e4e-a1ee-690b494fda35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql"
Oct 30 23:28:08.912466 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:08.912319    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmrcf\" (UniqueName: \"kubernetes.io/projected/0fabb604-c8a4-4e4e-a1ee-690b494fda35-kube-api-access-lmrcf\") pod \"maas-api-revoke-sub-34ef737b-fktql\" (UID: \"0fabb604-c8a4-4e4e-a1ee-690b494fda35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql"
Oct 30 23:28:08.912534 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:08.912516    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fabb604-c8a4-4e4e-a1ee-690b494fda35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34ef737b-fktql\" (UID: \"0fabb604-c8a4-4e4e-a1ee-690b494fda35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql"
Oct 30 23:28:08.913137 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:08.913115    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fabb604-c8a4-4e4e-a1ee-690b494fda35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34ef737b-fktql\" (UID: \"0fabb604-c8a4-4e4e-a1ee-690b494fda35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql"
Oct 30 23:28:08.914783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:08.914766    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fabb604-c8a4-4e4e-a1ee-690b494fda35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34ef737b-fktql\" (UID: \"0fabb604-c8a4-4e4e-a1ee-690b494fda35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql"
Oct 30 23:28:08.969388 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:08.969322    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmrcf\" (UniqueName: \"kubernetes.io/projected/0fabb604-c8a4-4e4e-a1ee-690b494fda35-kube-api-access-lmrcf\") pod \"maas-api-revoke-sub-34ef737b-fktql\" (UID: \"0fabb604-c8a4-4e4e-a1ee-690b494fda35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql"
Oct 30 23:28:09.032139 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.032103    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql"
Oct 30 23:28:09.049437 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.049396    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q"]
Oct 30 23:28:09.054214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.054188    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3dde47e9-vnc8q"]
Oct 30 23:28:09.113875 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.113530    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fbxsv-m7vqv"
Oct 30 23:28:09.216928 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.216880    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5465b09b-ab0c-462e-9e44-c3f884253bb6-kube-api-access-kj98z\" (UniqueName: \"kubernetes.io/projected/5465b09b-ab0c-462e-9e44-c3f884253bb6-kube-api-access-kj98z\") pod \"5465b09b-ab0c-462e-9e44-c3f884253bb6\" (UID: \"5465b09b-ab0c-462e-9e44-c3f884253bb6\") "
Oct 30 23:28:09.217115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.217028    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5465b09b-ab0c-462e-9e44-c3f884253bb6-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5465b09b-ab0c-462e-9e44-c3f884253bb6-archives-path\") pod \"5465b09b-ab0c-462e-9e44-c3f884253bb6\" (UID: \"5465b09b-ab0c-462e-9e44-c3f884253bb6\") "
Oct 30 23:28:09.217115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.217070    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5465b09b-ab0c-462e-9e44-c3f884253bb6-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5465b09b-ab0c-462e-9e44-c3f884253bb6-trusted-ca-bundle\") pod \"5465b09b-ab0c-462e-9e44-c3f884253bb6\" (UID: \"5465b09b-ab0c-462e-9e44-c3f884253bb6\") "
Oct 30 23:28:09.217246 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.217120    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5465b09b-ab0c-462e-9e44-c3f884253bb6-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5465b09b-ab0c-462e-9e44-c3f884253bb6-service-ca-bundle\") pod \"5465b09b-ab0c-462e-9e44-c3f884253bb6\" (UID: \"5465b09b-ab0c-462e-9e44-c3f884253bb6\") "
Oct 30 23:28:09.217484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.217414    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5465b09b-ab0c-462e-9e44-c3f884253bb6-archives-path" pod "5465b09b-ab0c-462e-9e44-c3f884253bb6" (UID: "5465b09b-ab0c-462e-9e44-c3f884253bb6"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:28:09.217572 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.217496    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5465b09b-ab0c-462e-9e44-c3f884253bb6-trusted-ca-bundle" pod "5465b09b-ab0c-462e-9e44-c3f884253bb6" (UID: "5465b09b-ab0c-462e-9e44-c3f884253bb6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:28:09.217572 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.217560    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5465b09b-ab0c-462e-9e44-c3f884253bb6-service-ca-bundle" pod "5465b09b-ab0c-462e-9e44-c3f884253bb6" (UID: "5465b09b-ab0c-462e-9e44-c3f884253bb6"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:28:09.217706 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.217653    5078 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/5465b09b-ab0c-462e-9e44-c3f884253bb6-archives-path\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:09.217706 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.217673    5078 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5465b09b-ab0c-462e-9e44-c3f884253bb6-trusted-ca-bundle\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:09.219131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.219112    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5465b09b-ab0c-462e-9e44-c3f884253bb6-kube-api-access-kj98z" pod "5465b09b-ab0c-462e-9e44-c3f884253bb6" (UID: "5465b09b-ab0c-462e-9e44-c3f884253bb6"). InnerVolumeSpecName "kube-api-access-kj98z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:28:09.319007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.318967    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj98z\" (UniqueName: \"kubernetes.io/projected/5465b09b-ab0c-462e-9e44-c3f884253bb6-kube-api-access-kj98z\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:09.319007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.319008    5078 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5465b09b-ab0c-462e-9e44-c3f884253bb6-service-ca-bundle\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:09.415022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.414994    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql"]
Oct 30 23:28:09.415730 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:28:09.415704    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fabb604_c8a4_4e4e_a1ee_690b494fda35.slice/crio-6a8092fd15934648f16f249f642190410534abbd50abf67b284a2c24199222fd WatchSource:0}: Error finding container 6a8092fd15934648f16f249f642190410534abbd50abf67b284a2c24199222fd: Status 404 returned error can't find the container with id 6a8092fd15934648f16f249f642190410534abbd50abf67b284a2c24199222fd
Oct 30 23:28:09.950340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.950303    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql" event={"ID":"0fabb604-c8a4-4e4e-a1ee-690b494fda35","Type":"ContainerStarted","Data":"a05bbd305c6199488691103303a37d90fd09ec04836213e3718a7b45f1cbe1d9"}
Oct 30 23:28:09.950786 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.950367    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql" event={"ID":"0fabb604-c8a4-4e4e-a1ee-690b494fda35","Type":"ContainerStarted","Data":"6a8092fd15934648f16f249f642190410534abbd50abf67b284a2c24199222fd"}
Oct 30 23:28:09.951869 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.951849    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-fbxsv-m7vqv"
Oct 30 23:28:09.952002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.951894    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-fbxsv-m7vqv" event={"ID":"5465b09b-ab0c-462e-9e44-c3f884253bb6","Type":"ContainerDied","Data":"de3d0fe4ade8911955b2c8344ff8d7b0e9015c414be7117cdc381c8258fcbf28"}
Oct 30 23:28:09.952002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:09.951928    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de3d0fe4ade8911955b2c8344ff8d7b0e9015c414be7117cdc381c8258fcbf28"
Oct 30 23:28:10.068089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.068038    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql" podStartSLOduration=2.068021904 podStartE2EDuration="2.068021904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:28:10.065720823 +0000 UTC m=+1275.953627179" watchObservedRunningTime="2026-10-30 23:28:10.068021904 +0000 UTC m=+1275.955928257"
Oct 30 23:28:10.299420 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.299388    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="156a09d7-39fb-4f38-964d-bc0d7ccaa341" path="/var/lib/kubelet/pods/156a09d7-39fb-4f38-964d-bc0d7ccaa341/volumes"
Oct 30 23:28:10.301906 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.301878    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl"]
Oct 30 23:28:10.302448 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.302429    5078 cpu_manager.go:414] "containerMap: removing container" podUID="5465b09b-ab0c-462e-9e44-c3f884253bb6" containerName="insights-gathering"
Oct 30 23:28:10.302539 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.302450    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5465b09b-ab0c-462e-9e44-c3f884253bb6" containerName="insights-gathering"
Oct 30 23:28:10.302677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.302662    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="5465b09b-ab0c-462e-9e44-c3f884253bb6" containerName="insights-gathering"
Oct 30 23:28:10.306160 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.306143    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl"
Oct 30 23:28:10.361956 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.361932    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl"]
Oct 30 23:28:10.431523 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.431494    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-356b0f8d-qgmpl\" (UID: \"de16dc9d-9d25-4439-9b64-3f40d5e2bc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl"
Oct 30 23:28:10.431690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.431527    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-356b0f8d-qgmpl\" (UID: \"de16dc9d-9d25-4439-9b64-3f40d5e2bc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl"
Oct 30 23:28:10.431690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.431578    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbghr\" (UniqueName: \"kubernetes.io/projected/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-kube-api-access-vbghr\") pod \"maas-api-revoke-sub-356b0f8d-qgmpl\" (UID: \"de16dc9d-9d25-4439-9b64-3f40d5e2bc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl"
Oct 30 23:28:10.532862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.532830    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-356b0f8d-qgmpl\" (UID: \"de16dc9d-9d25-4439-9b64-3f40d5e2bc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl"
Oct 30 23:28:10.533057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.532871    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-356b0f8d-qgmpl\" (UID: \"de16dc9d-9d25-4439-9b64-3f40d5e2bc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl"
Oct 30 23:28:10.533057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.532904    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbghr\" (UniqueName: \"kubernetes.io/projected/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-kube-api-access-vbghr\") pod \"maas-api-revoke-sub-356b0f8d-qgmpl\" (UID: \"de16dc9d-9d25-4439-9b64-3f40d5e2bc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl"
Oct 30 23:28:10.533496 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.533472    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-356b0f8d-qgmpl\" (UID: \"de16dc9d-9d25-4439-9b64-3f40d5e2bc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl"
Oct 30 23:28:10.535454 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.535436    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-356b0f8d-qgmpl\" (UID: \"de16dc9d-9d25-4439-9b64-3f40d5e2bc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl"
Oct 30 23:28:10.590074 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.590014    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbghr\" (UniqueName: \"kubernetes.io/projected/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-kube-api-access-vbghr\") pod \"maas-api-revoke-sub-356b0f8d-qgmpl\" (UID: \"de16dc9d-9d25-4439-9b64-3f40d5e2bc70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl"
Oct 30 23:28:10.619223 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.619202    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl"
Oct 30 23:28:10.959750 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.959662    5078 generic.go:356] "Generic (PLEG): container finished" podID="0fabb604-c8a4-4e4e-a1ee-690b494fda35" containerID="a05bbd305c6199488691103303a37d90fd09ec04836213e3718a7b45f1cbe1d9" exitCode=0
Oct 30 23:28:10.960127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.959754    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql" event={"ID":"0fabb604-c8a4-4e4e-a1ee-690b494fda35","Type":"ContainerDied","Data":"a05bbd305c6199488691103303a37d90fd09ec04836213e3718a7b45f1cbe1d9"}
Oct 30 23:28:10.993984 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:10.993957    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl"]
Oct 30 23:28:10.994750 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:28:10.994722    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde16dc9d_9d25_4439_9b64_3f40d5e2bc70.slice/crio-200dad1d90b0b24135fa6ce9002621a6f87b1c92e3dcb60eb1a4fdb5ea6a881a WatchSource:0}: Error finding container 200dad1d90b0b24135fa6ce9002621a6f87b1c92e3dcb60eb1a4fdb5ea6a881a: Status 404 returned error can't find the container with id 200dad1d90b0b24135fa6ce9002621a6f87b1c92e3dcb60eb1a4fdb5ea6a881a
Oct 30 23:28:11.967876 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:11.967840    5078 generic.go:356] "Generic (PLEG): container finished" podID="de16dc9d-9d25-4439-9b64-3f40d5e2bc70" containerID="cad926f3da76b32be863ace00eb465d02b876c2324525f07c4bea6cc88feac08" exitCode=0
Oct 30 23:28:11.968290 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:11.967925    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl" event={"ID":"de16dc9d-9d25-4439-9b64-3f40d5e2bc70","Type":"ContainerDied","Data":"cad926f3da76b32be863ace00eb465d02b876c2324525f07c4bea6cc88feac08"}
Oct 30 23:28:11.968290 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:11.967974    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl" event={"ID":"de16dc9d-9d25-4439-9b64-3f40d5e2bc70","Type":"ContainerStarted","Data":"200dad1d90b0b24135fa6ce9002621a6f87b1c92e3dcb60eb1a4fdb5ea6a881a"}
Oct 30 23:28:12.117008 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:12.116979    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql"
Oct 30 23:28:12.258098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:12.258070    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fabb604-c8a4-4e4e-a1ee-690b494fda35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fabb604-c8a4-4e4e-a1ee-690b494fda35-maas-api-xks-ca\") pod \"0fabb604-c8a4-4e4e-a1ee-690b494fda35\" (UID: \"0fabb604-c8a4-4e4e-a1ee-690b494fda35\") "
Oct 30 23:28:12.258265 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:12.258176    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fabb604-c8a4-4e4e-a1ee-690b494fda35-kube-api-access-lmrcf\" (UniqueName: \"kubernetes.io/projected/0fabb604-c8a4-4e4e-a1ee-690b494fda35-kube-api-access-lmrcf\") pod \"0fabb604-c8a4-4e4e-a1ee-690b494fda35\" (UID: \"0fabb604-c8a4-4e4e-a1ee-690b494fda35\") "
Oct 30 23:28:12.258363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:12.258331    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fabb604-c8a4-4e4e-a1ee-690b494fda35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fabb604-c8a4-4e4e-a1ee-690b494fda35-maas-api-openshift-ca\") pod \"0fabb604-c8a4-4e4e-a1ee-690b494fda35\" (UID: \"0fabb604-c8a4-4e4e-a1ee-690b494fda35\") "
Oct 30 23:28:12.258762 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:12.258730    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fabb604-c8a4-4e4e-a1ee-690b494fda35-maas-api-openshift-ca" pod "0fabb604-c8a4-4e4e-a1ee-690b494fda35" (UID: "0fabb604-c8a4-4e4e-a1ee-690b494fda35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:28:12.259023 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:12.259004    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fabb604-c8a4-4e4e-a1ee-690b494fda35-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:12.260870 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:12.260831    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fabb604-c8a4-4e4e-a1ee-690b494fda35-kube-api-access-lmrcf" pod "0fabb604-c8a4-4e4e-a1ee-690b494fda35" (UID: "0fabb604-c8a4-4e4e-a1ee-690b494fda35"). InnerVolumeSpecName "kube-api-access-lmrcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:28:12.261013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:12.260990    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fabb604-c8a4-4e4e-a1ee-690b494fda35-maas-api-xks-ca" pod "0fabb604-c8a4-4e4e-a1ee-690b494fda35" (UID: "0fabb604-c8a4-4e4e-a1ee-690b494fda35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:28:12.360789 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:12.360715    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fabb604-c8a4-4e4e-a1ee-690b494fda35-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:12.360789 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:12.360748    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmrcf\" (UniqueName: \"kubernetes.io/projected/0fabb604-c8a4-4e4e-a1ee-690b494fda35-kube-api-access-lmrcf\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:12.975966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:12.975932    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql"
Oct 30 23:28:12.975966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:12.975950    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql" event={"ID":"0fabb604-c8a4-4e4e-a1ee-690b494fda35","Type":"ContainerDied","Data":"6a8092fd15934648f16f249f642190410534abbd50abf67b284a2c24199222fd"}
Oct 30 23:28:12.976490 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:12.976000    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a8092fd15934648f16f249f642190410534abbd50abf67b284a2c24199222fd"
Oct 30 23:28:13.127414 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:13.127385    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl"
Oct 30 23:28:13.275660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:13.275527    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-kube-api-access-vbghr\" (UniqueName: \"kubernetes.io/projected/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-kube-api-access-vbghr\") pod \"de16dc9d-9d25-4439-9b64-3f40d5e2bc70\" (UID: \"de16dc9d-9d25-4439-9b64-3f40d5e2bc70\") "
Oct 30 23:28:13.275660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:13.275638    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-maas-api-openshift-ca\") pod \"de16dc9d-9d25-4439-9b64-3f40d5e2bc70\" (UID: \"de16dc9d-9d25-4439-9b64-3f40d5e2bc70\") "
Oct 30 23:28:13.275912 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:13.275823    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-maas-api-xks-ca\") pod \"de16dc9d-9d25-4439-9b64-3f40d5e2bc70\" (UID: \"de16dc9d-9d25-4439-9b64-3f40d5e2bc70\") "
Oct 30 23:28:13.276044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:13.276014    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-maas-api-openshift-ca" pod "de16dc9d-9d25-4439-9b64-3f40d5e2bc70" (UID: "de16dc9d-9d25-4439-9b64-3f40d5e2bc70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:28:13.276648 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:13.276621    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:13.278032 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:13.278005    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-maas-api-xks-ca" pod "de16dc9d-9d25-4439-9b64-3f40d5e2bc70" (UID: "de16dc9d-9d25-4439-9b64-3f40d5e2bc70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:28:13.278351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:13.278330    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-kube-api-access-vbghr" pod "de16dc9d-9d25-4439-9b64-3f40d5e2bc70" (UID: "de16dc9d-9d25-4439-9b64-3f40d5e2bc70"). InnerVolumeSpecName "kube-api-access-vbghr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:28:13.378769 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:13.378682    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:13.378769 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:13.378723    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbghr\" (UniqueName: \"kubernetes.io/projected/de16dc9d-9d25-4439-9b64-3f40d5e2bc70-kube-api-access-vbghr\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:14.038808 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:14.038771    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl" event={"ID":"de16dc9d-9d25-4439-9b64-3f40d5e2bc70","Type":"ContainerDied","Data":"200dad1d90b0b24135fa6ce9002621a6f87b1c92e3dcb60eb1a4fdb5ea6a881a"}
Oct 30 23:28:14.039361 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:14.039338    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="200dad1d90b0b24135fa6ce9002621a6f87b1c92e3dcb60eb1a4fdb5ea6a881a"
Oct 30 23:28:14.039582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:14.039560    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl"
Oct 30 23:28:15.284740 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:15.284702    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:28:15.476759 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:15.476680    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:28:15.519434 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:15.518682    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:28:15.555753 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:15.555714    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:28:15.662740 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:15.662655    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:28:16.584653 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.581527    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96"]
Oct 30 23:28:16.608129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.607855    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0fabb604-c8a4-4e4e-a1ee-690b494fda35" containerName="revoke-keys"
Oct 30 23:28:16.608129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.607883    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fabb604-c8a4-4e4e-a1ee-690b494fda35" containerName="revoke-keys"
Oct 30 23:28:16.608129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.607912    5078 cpu_manager.go:414] "containerMap: removing container" podUID="de16dc9d-9d25-4439-9b64-3f40d5e2bc70" containerName="revoke-keys"
Oct 30 23:28:16.608129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.607921    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de16dc9d-9d25-4439-9b64-3f40d5e2bc70" containerName="revoke-keys"
Oct 30 23:28:16.608129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.608081    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="de16dc9d-9d25-4439-9b64-3f40d5e2bc70" containerName="revoke-keys"
Oct 30 23:28:16.608129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.608106    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fabb604-c8a4-4e4e-a1ee-690b494fda35" containerName="revoke-keys"
Oct 30 23:28:16.618008 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.617978    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96"
Oct 30 23:28:16.635886 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.635856    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2159d7-8592-4372-830f-e791ce996831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07aadec7-7mm96\" (UID: \"ad2159d7-8592-4372-830f-e791ce996831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96"
Oct 30 23:28:16.636153 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.636134    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zvvj\" (UniqueName: \"kubernetes.io/projected/ad2159d7-8592-4372-830f-e791ce996831-kube-api-access-5zvvj\") pod \"maas-api-revoke-sub-07aadec7-7mm96\" (UID: \"ad2159d7-8592-4372-830f-e791ce996831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96"
Oct 30 23:28:16.636405 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.636390    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2159d7-8592-4372-830f-e791ce996831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07aadec7-7mm96\" (UID: \"ad2159d7-8592-4372-830f-e791ce996831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96"
Oct 30 23:28:16.642058 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.642033    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96"]
Oct 30 23:28:16.683881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.683852    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:28:16.684040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.683907    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:28:16.684040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.683930    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:28:16.720401 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.720378    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:28:16.738290 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.738267    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2159d7-8592-4372-830f-e791ce996831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07aadec7-7mm96\" (UID: \"ad2159d7-8592-4372-830f-e791ce996831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96"
Oct 30 23:28:16.738428 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.738350    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2159d7-8592-4372-830f-e791ce996831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07aadec7-7mm96\" (UID: \"ad2159d7-8592-4372-830f-e791ce996831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96"
Oct 30 23:28:16.738428 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.738394    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvvj\" (UniqueName: \"kubernetes.io/projected/ad2159d7-8592-4372-830f-e791ce996831-kube-api-access-5zvvj\") pod \"maas-api-revoke-sub-07aadec7-7mm96\" (UID: \"ad2159d7-8592-4372-830f-e791ce996831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96"
Oct 30 23:28:16.739013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.738989    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2159d7-8592-4372-830f-e791ce996831-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07aadec7-7mm96\" (UID: \"ad2159d7-8592-4372-830f-e791ce996831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96"
Oct 30 23:28:16.740949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.740906    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2159d7-8592-4372-830f-e791ce996831-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07aadec7-7mm96\" (UID: \"ad2159d7-8592-4372-830f-e791ce996831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96"
Oct 30 23:28:16.795111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.795084    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zvvj\" (UniqueName: \"kubernetes.io/projected/ad2159d7-8592-4372-830f-e791ce996831-kube-api-access-5zvvj\") pod \"maas-api-revoke-sub-07aadec7-7mm96\" (UID: \"ad2159d7-8592-4372-830f-e791ce996831\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96"
Oct 30 23:28:16.945715 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:16.945637    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96"
Oct 30 23:28:17.334132 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:17.334095    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96"]
Oct 30 23:28:17.334285 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:28:17.334222    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2159d7_8592_4372_830f_e791ce996831.slice/crio-e61da7ebc2592fd7dc315cdb98444ff2a2c401a21962e134bc43ee2f4b153ffd WatchSource:0}: Error finding container e61da7ebc2592fd7dc315cdb98444ff2a2c401a21962e134bc43ee2f4b153ffd: Status 404 returned error can't find the container with id e61da7ebc2592fd7dc315cdb98444ff2a2c401a21962e134bc43ee2f4b153ffd
Oct 30 23:28:17.592429 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:28:17.592351    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:28:18.078343 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:18.078308    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96" event={"ID":"ad2159d7-8592-4372-830f-e791ce996831","Type":"ContainerStarted","Data":"6d3f35723c3a0b747d1ec5b5bc479f14488373f2aed275300ec24bdaefb5777e"}
Oct 30 23:28:18.078528 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:18.078366    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96" event={"ID":"ad2159d7-8592-4372-830f-e791ce996831","Type":"ContainerStarted","Data":"e61da7ebc2592fd7dc315cdb98444ff2a2c401a21962e134bc43ee2f4b153ffd"}
Oct 30 23:28:18.195146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:18.195095    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96" podStartSLOduration=2.195082529 podStartE2EDuration="2.195082529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:28:18.193154273 +0000 UTC m=+1284.081060627" watchObservedRunningTime="2026-10-30 23:28:18.195082529 +0000 UTC m=+1284.082988881"
Oct 30 23:28:19.086026 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:19.085994    5078 generic.go:356] "Generic (PLEG): container finished" podID="ad2159d7-8592-4372-830f-e791ce996831" containerID="6d3f35723c3a0b747d1ec5b5bc479f14488373f2aed275300ec24bdaefb5777e" exitCode=0
Oct 30 23:28:19.086469 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:19.086071    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96" event={"ID":"ad2159d7-8592-4372-830f-e791ce996831","Type":"ContainerDied","Data":"6d3f35723c3a0b747d1ec5b5bc479f14488373f2aed275300ec24bdaefb5777e"}
Oct 30 23:28:20.047100 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:20.047068    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22"]
Oct 30 23:28:20.096661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:20.096638    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c970766b-dzj22"]
Oct 30 23:28:20.251015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:20.250991    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96"
Oct 30 23:28:20.284345 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:20.284314    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad2159d7-8592-4372-830f-e791ce996831-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2159d7-8592-4372-830f-e791ce996831-maas-api-openshift-ca\") pod \"ad2159d7-8592-4372-830f-e791ce996831\" (UID: \"ad2159d7-8592-4372-830f-e791ce996831\") "
Oct 30 23:28:20.284567 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:20.284551    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad2159d7-8592-4372-830f-e791ce996831-kube-api-access-5zvvj\" (UniqueName: \"kubernetes.io/projected/ad2159d7-8592-4372-830f-e791ce996831-kube-api-access-5zvvj\") pod \"ad2159d7-8592-4372-830f-e791ce996831\" (UID: \"ad2159d7-8592-4372-830f-e791ce996831\") "
Oct 30 23:28:20.284678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:20.284624    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad2159d7-8592-4372-830f-e791ce996831-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2159d7-8592-4372-830f-e791ce996831-maas-api-xks-ca\") pod \"ad2159d7-8592-4372-830f-e791ce996831\" (UID: \"ad2159d7-8592-4372-830f-e791ce996831\") "
Oct 30 23:28:20.284678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:20.284644    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2159d7-8592-4372-830f-e791ce996831-maas-api-openshift-ca" pod "ad2159d7-8592-4372-830f-e791ce996831" (UID: "ad2159d7-8592-4372-830f-e791ce996831"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:28:20.285104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:20.285084    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad2159d7-8592-4372-830f-e791ce996831-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:20.286658 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:20.286630    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad2159d7-8592-4372-830f-e791ce996831-maas-api-xks-ca" pod "ad2159d7-8592-4372-830f-e791ce996831" (UID: "ad2159d7-8592-4372-830f-e791ce996831"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:28:20.286753 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:20.286699    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2159d7-8592-4372-830f-e791ce996831-kube-api-access-5zvvj" pod "ad2159d7-8592-4372-830f-e791ce996831" (UID: "ad2159d7-8592-4372-830f-e791ce996831"). InnerVolumeSpecName "kube-api-access-5zvvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:28:20.298696 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:20.298646    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afa22026-5e03-4295-8dd6-44b21b694159" path="/var/lib/kubelet/pods/afa22026-5e03-4295-8dd6-44b21b694159/volumes"
Oct 30 23:28:20.386727 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:20.386676    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zvvj\" (UniqueName: \"kubernetes.io/projected/ad2159d7-8592-4372-830f-e791ce996831-kube-api-access-5zvvj\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:20.386727 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:20.386710    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad2159d7-8592-4372-830f-e791ce996831-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:21.101728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.101700    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96"
Oct 30 23:28:21.102154 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.101691    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96" event={"ID":"ad2159d7-8592-4372-830f-e791ce996831","Type":"ContainerDied","Data":"e61da7ebc2592fd7dc315cdb98444ff2a2c401a21962e134bc43ee2f4b153ffd"}
Oct 30 23:28:21.102154 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.101831    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e61da7ebc2592fd7dc315cdb98444ff2a2c401a21962e134bc43ee2f4b153ffd"
Oct 30 23:28:21.456710 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.456632    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw"]
Oct 30 23:28:21.457156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.457143    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ad2159d7-8592-4372-830f-e791ce996831" containerName="revoke-keys"
Oct 30 23:28:21.457204 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.457157    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2159d7-8592-4372-830f-e791ce996831" containerName="revoke-keys"
Oct 30 23:28:21.457285 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.457276    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2159d7-8592-4372-830f-e791ce996831" containerName="revoke-keys"
Oct 30 23:28:21.460996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.460979    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw"
Oct 30 23:28:21.500725 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.500701    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11406c63-9c89-460f-bb55-1af3b29bacda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-752f525b-8wllw\" (UID: \"11406c63-9c89-460f-bb55-1af3b29bacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw"
Oct 30 23:28:21.500862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.500769    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11406c63-9c89-460f-bb55-1af3b29bacda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-752f525b-8wllw\" (UID: \"11406c63-9c89-460f-bb55-1af3b29bacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw"
Oct 30 23:28:21.500920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.500899    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znw9b\" (UniqueName: \"kubernetes.io/projected/11406c63-9c89-460f-bb55-1af3b29bacda-kube-api-access-znw9b\") pod \"maas-api-revoke-sub-752f525b-8wllw\" (UID: \"11406c63-9c89-460f-bb55-1af3b29bacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw"
Oct 30 23:28:21.513478 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.513452    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:28:21.513765 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.513534    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:28:21.513765 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.513646    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:28:21.517142 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.517121    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw"]
Oct 30 23:28:21.561498 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.561467    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:28:21.602023 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.601991    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11406c63-9c89-460f-bb55-1af3b29bacda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-752f525b-8wllw\" (UID: \"11406c63-9c89-460f-bb55-1af3b29bacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw"
Oct 30 23:28:21.602163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.602043    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11406c63-9c89-460f-bb55-1af3b29bacda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-752f525b-8wllw\" (UID: \"11406c63-9c89-460f-bb55-1af3b29bacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw"
Oct 30 23:28:21.602163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.602108    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znw9b\" (UniqueName: \"kubernetes.io/projected/11406c63-9c89-460f-bb55-1af3b29bacda-kube-api-access-znw9b\") pod \"maas-api-revoke-sub-752f525b-8wllw\" (UID: \"11406c63-9c89-460f-bb55-1af3b29bacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw"
Oct 30 23:28:21.602735 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.602712    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11406c63-9c89-460f-bb55-1af3b29bacda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-752f525b-8wllw\" (UID: \"11406c63-9c89-460f-bb55-1af3b29bacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw"
Oct 30 23:28:21.604729 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.604709    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11406c63-9c89-460f-bb55-1af3b29bacda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-752f525b-8wllw\" (UID: \"11406c63-9c89-460f-bb55-1af3b29bacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw"
Oct 30 23:28:21.658834 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.658808    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znw9b\" (UniqueName: \"kubernetes.io/projected/11406c63-9c89-460f-bb55-1af3b29bacda-kube-api-access-znw9b\") pod \"maas-api-revoke-sub-752f525b-8wllw\" (UID: \"11406c63-9c89-460f-bb55-1af3b29bacda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw"
Oct 30 23:28:21.775538 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:21.775467    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw"
Oct 30 23:28:22.152314 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:28:22.152283    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11406c63_9c89_460f_bb55_1af3b29bacda.slice/crio-6c3d49dec768f80f190e00d8b158f8bd8a09661611436c659e50c4b02b340dde WatchSource:0}: Error finding container 6c3d49dec768f80f190e00d8b158f8bd8a09661611436c659e50c4b02b340dde: Status 404 returned error can't find the container with id 6c3d49dec768f80f190e00d8b158f8bd8a09661611436c659e50c4b02b340dde
Oct 30 23:28:22.153481 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:22.153456    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw"]
Oct 30 23:28:23.118463 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:23.118436    5078 generic.go:356] "Generic (PLEG): container finished" podID="11406c63-9c89-460f-bb55-1af3b29bacda" containerID="24c856a20e568c3eff2bebe8765c5c38fe2e0cc431e9229ba4050c5dda957d35" exitCode=0
Oct 30 23:28:23.118625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:23.118532    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw" event={"ID":"11406c63-9c89-460f-bb55-1af3b29bacda","Type":"ContainerDied","Data":"24c856a20e568c3eff2bebe8765c5c38fe2e0cc431e9229ba4050c5dda957d35"}
Oct 30 23:28:23.118625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:23.118589    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw" event={"ID":"11406c63-9c89-460f-bb55-1af3b29bacda","Type":"ContainerStarted","Data":"6c3d49dec768f80f190e00d8b158f8bd8a09661611436c659e50c4b02b340dde"}
Oct 30 23:28:24.047761 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:24.047728    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt"]
Oct 30 23:28:24.097935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:24.097906    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3f37608-g8pqt"]
Oct 30 23:28:24.295241 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:24.295213    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw"
Oct 30 23:28:24.298779 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:24.298727    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c655ce-f2ca-411a-b95e-48adcd4fe77e" path="/var/lib/kubelet/pods/d5c655ce-f2ca-411a-b95e-48adcd4fe77e/volumes"
Oct 30 23:28:24.454359 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:24.454323    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11406c63-9c89-460f-bb55-1af3b29bacda-kube-api-access-znw9b\" (UniqueName: \"kubernetes.io/projected/11406c63-9c89-460f-bb55-1af3b29bacda-kube-api-access-znw9b\") pod \"11406c63-9c89-460f-bb55-1af3b29bacda\" (UID: \"11406c63-9c89-460f-bb55-1af3b29bacda\") "
Oct 30 23:28:24.454550 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:24.454388    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11406c63-9c89-460f-bb55-1af3b29bacda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11406c63-9c89-460f-bb55-1af3b29bacda-maas-api-xks-ca\") pod \"11406c63-9c89-460f-bb55-1af3b29bacda\" (UID: \"11406c63-9c89-460f-bb55-1af3b29bacda\") "
Oct 30 23:28:24.454550 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:24.454515    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11406c63-9c89-460f-bb55-1af3b29bacda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11406c63-9c89-460f-bb55-1af3b29bacda-maas-api-openshift-ca\") pod \"11406c63-9c89-460f-bb55-1af3b29bacda\" (UID: \"11406c63-9c89-460f-bb55-1af3b29bacda\") "
Oct 30 23:28:24.454880 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:24.454845    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11406c63-9c89-460f-bb55-1af3b29bacda-maas-api-openshift-ca" pod "11406c63-9c89-460f-bb55-1af3b29bacda" (UID: "11406c63-9c89-460f-bb55-1af3b29bacda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:28:24.455503 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:24.455234    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11406c63-9c89-460f-bb55-1af3b29bacda-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:24.456799 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:24.456778    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11406c63-9c89-460f-bb55-1af3b29bacda-kube-api-access-znw9b" pod "11406c63-9c89-460f-bb55-1af3b29bacda" (UID: "11406c63-9c89-460f-bb55-1af3b29bacda"). InnerVolumeSpecName "kube-api-access-znw9b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:28:24.456902 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:24.456826    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11406c63-9c89-460f-bb55-1af3b29bacda-maas-api-xks-ca" pod "11406c63-9c89-460f-bb55-1af3b29bacda" (UID: "11406c63-9c89-460f-bb55-1af3b29bacda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:28:24.556551 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:24.556484    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znw9b\" (UniqueName: \"kubernetes.io/projected/11406c63-9c89-460f-bb55-1af3b29bacda-kube-api-access-znw9b\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:24.556551 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:24.556511    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11406c63-9c89-460f-bb55-1af3b29bacda-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:25.133831 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:25.133805    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw"
Oct 30 23:28:25.134221 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:25.133849    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw" event={"ID":"11406c63-9c89-460f-bb55-1af3b29bacda","Type":"ContainerDied","Data":"6c3d49dec768f80f190e00d8b158f8bd8a09661611436c659e50c4b02b340dde"}
Oct 30 23:28:25.134221 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:25.133902    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c3d49dec768f80f190e00d8b158f8bd8a09661611436c659e50c4b02b340dde"
Oct 30 23:28:28.049065 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:28.049031    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m"]
Oct 30 23:28:28.098512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:28.098471    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a45668e-prx6m"]
Oct 30 23:28:28.298961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:28.298927    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98a11942-14ba-4757-8264-01bf9d03db99" path="/var/lib/kubelet/pods/98a11942-14ba-4757-8264-01bf9d03db99/volumes"
Oct 30 23:28:31.809128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:31.809095    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd"]
Oct 30 23:28:31.809658 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:31.809642    5078 cpu_manager.go:414] "containerMap: removing container" podUID="11406c63-9c89-460f-bb55-1af3b29bacda" containerName="revoke-keys"
Oct 30 23:28:31.809658 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:31.809659    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11406c63-9c89-460f-bb55-1af3b29bacda" containerName="revoke-keys"
Oct 30 23:28:31.809778 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:31.809769    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="11406c63-9c89-460f-bb55-1af3b29bacda" containerName="revoke-keys"
Oct 30 23:28:31.814437 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:31.814418    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd"
Oct 30 23:28:31.865801 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:31.865765    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb9s8\" (UniqueName: \"kubernetes.io/projected/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-kube-api-access-hb9s8\") pod \"maas-api-revoke-sub-0a89059e-sqbxd\" (UID: \"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd"
Oct 30 23:28:31.866000 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:31.865976    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a89059e-sqbxd\" (UID: \"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd"
Oct 30 23:28:31.866000 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:31.866010    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a89059e-sqbxd\" (UID: \"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd"
Oct 30 23:28:31.869064 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:31.869042    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd"]
Oct 30 23:28:31.913861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:31.913833    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:28:31.913861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:31.913833    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:28:31.913861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:31.913863    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:28:31.915895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:31.915877    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:28:31.967043 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:31.967014    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb9s8\" (UniqueName: \"kubernetes.io/projected/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-kube-api-access-hb9s8\") pod \"maas-api-revoke-sub-0a89059e-sqbxd\" (UID: \"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd"
Oct 30 23:28:31.967168 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:31.967052    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a89059e-sqbxd\" (UID: \"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd"
Oct 30 23:28:31.967168 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:31.967070    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a89059e-sqbxd\" (UID: \"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd"
Oct 30 23:28:31.967723 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:31.967706    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a89059e-sqbxd\" (UID: \"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd"
Oct 30 23:28:31.969575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:31.969553    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a89059e-sqbxd\" (UID: \"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd"
Oct 30 23:28:32.023819 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:32.023791    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb9s8\" (UniqueName: \"kubernetes.io/projected/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-kube-api-access-hb9s8\") pod \"maas-api-revoke-sub-0a89059e-sqbxd\" (UID: \"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd"
Oct 30 23:28:32.127797 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:32.127714    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd"
Oct 30 23:28:32.505795 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:28:32.505766    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0a87d8_687d_4ebf_bd66_aa6cc59701b0.slice/crio-0c6ad2168f20df2338381c769f701d65a5559e5786f715a99ec024faf49632ae WatchSource:0}: Error finding container 0c6ad2168f20df2338381c769f701d65a5559e5786f715a99ec024faf49632ae: Status 404 returned error can't find the container with id 0c6ad2168f20df2338381c769f701d65a5559e5786f715a99ec024faf49632ae
Oct 30 23:28:32.506547 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:32.506517    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd"]
Oct 30 23:28:33.068053 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.068022    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf"]
Oct 30 23:28:33.072208 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.072190    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf"
Oct 30 23:28:33.128839 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.128809    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf"]
Oct 30 23:28:33.183791 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.183759    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8f824de-6ac1-491a-abc3-d4a941873bdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-576b45a7-bjmvf\" (UID: \"b8f824de-6ac1-491a-abc3-d4a941873bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf"
Oct 30 23:28:33.183936 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.183800    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8f824de-6ac1-491a-abc3-d4a941873bdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-576b45a7-bjmvf\" (UID: \"b8f824de-6ac1-491a-abc3-d4a941873bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf"
Oct 30 23:28:33.183994 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.183976    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tch42\" (UniqueName: \"kubernetes.io/projected/b8f824de-6ac1-491a-abc3-d4a941873bdf-kube-api-access-tch42\") pod \"maas-api-revoke-sub-576b45a7-bjmvf\" (UID: \"b8f824de-6ac1-491a-abc3-d4a941873bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf"
Oct 30 23:28:33.218004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.217965    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd" event={"ID":"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0","Type":"ContainerStarted","Data":"5c146234b9514d86a5b35b61431b884d39ae0799fe6a3db786526b41fb7ca94d"}
Oct 30 23:28:33.218178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.218030    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd" event={"ID":"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0","Type":"ContainerStarted","Data":"0c6ad2168f20df2338381c769f701d65a5559e5786f715a99ec024faf49632ae"}
Oct 30 23:28:33.285611 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.285572    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8f824de-6ac1-491a-abc3-d4a941873bdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-576b45a7-bjmvf\" (UID: \"b8f824de-6ac1-491a-abc3-d4a941873bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf"
Oct 30 23:28:33.285794 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.285663    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8f824de-6ac1-491a-abc3-d4a941873bdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-576b45a7-bjmvf\" (UID: \"b8f824de-6ac1-491a-abc3-d4a941873bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf"
Oct 30 23:28:33.285860 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.285831    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tch42\" (UniqueName: \"kubernetes.io/projected/b8f824de-6ac1-491a-abc3-d4a941873bdf-kube-api-access-tch42\") pod \"maas-api-revoke-sub-576b45a7-bjmvf\" (UID: \"b8f824de-6ac1-491a-abc3-d4a941873bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf"
Oct 30 23:28:33.286311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.286289    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8f824de-6ac1-491a-abc3-d4a941873bdf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-576b45a7-bjmvf\" (UID: \"b8f824de-6ac1-491a-abc3-d4a941873bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf"
Oct 30 23:28:33.288268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.288248    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8f824de-6ac1-491a-abc3-d4a941873bdf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-576b45a7-bjmvf\" (UID: \"b8f824de-6ac1-491a-abc3-d4a941873bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf"
Oct 30 23:28:33.332982 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.332884    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd" podStartSLOduration=2.33287002 podStartE2EDuration="2.33287002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:28:33.330270402 +0000 UTC m=+1299.218176755" watchObservedRunningTime="2026-10-30 23:28:33.33287002 +0000 UTC m=+1299.220776401"
Oct 30 23:28:33.344899 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.344859    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tch42\" (UniqueName: \"kubernetes.io/projected/b8f824de-6ac1-491a-abc3-d4a941873bdf-kube-api-access-tch42\") pod \"maas-api-revoke-sub-576b45a7-bjmvf\" (UID: \"b8f824de-6ac1-491a-abc3-d4a941873bdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf"
Oct 30 23:28:33.386371 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.386346    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf"
Oct 30 23:28:33.761537 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.761507    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf"]
Oct 30 23:28:33.761854 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:28:33.761829    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f824de_6ac1_491a_abc3_d4a941873bdf.slice/crio-a8bd3c9fb490f21a0605216f8914ab180a759bb352c5588b883db1856322e496 WatchSource:0}: Error finding container a8bd3c9fb490f21a0605216f8914ab180a759bb352c5588b883db1856322e496: Status 404 returned error can't find the container with id a8bd3c9fb490f21a0605216f8914ab180a759bb352c5588b883db1856322e496
Oct 30 23:28:33.925720 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.925676    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs"]
Oct 30 23:28:33.931406 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.931382    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs"
Oct 30 23:28:33.985678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.985646    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs"]
Oct 30 23:28:33.996315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.996292    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496633ba-dc94-4a3c-a6c4-079a2d0234b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d88717da-tkgzs\" (UID: \"496633ba-dc94-4a3c-a6c4-079a2d0234b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs"
Oct 30 23:28:33.996422 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.996324    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd76w\" (UniqueName: \"kubernetes.io/projected/496633ba-dc94-4a3c-a6c4-079a2d0234b9-kube-api-access-gd76w\") pod \"maas-api-revoke-sub-d88717da-tkgzs\" (UID: \"496633ba-dc94-4a3c-a6c4-079a2d0234b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs"
Oct 30 23:28:33.996565 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:33.996546    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496633ba-dc94-4a3c-a6c4-079a2d0234b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d88717da-tkgzs\" (UID: \"496633ba-dc94-4a3c-a6c4-079a2d0234b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs"
Oct 30 23:28:34.097532 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:34.097508    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496633ba-dc94-4a3c-a6c4-079a2d0234b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d88717da-tkgzs\" (UID: \"496633ba-dc94-4a3c-a6c4-079a2d0234b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs"
Oct 30 23:28:34.097859 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:34.097650    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496633ba-dc94-4a3c-a6c4-079a2d0234b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d88717da-tkgzs\" (UID: \"496633ba-dc94-4a3c-a6c4-079a2d0234b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs"
Oct 30 23:28:34.097859 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:34.097678    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd76w\" (UniqueName: \"kubernetes.io/projected/496633ba-dc94-4a3c-a6c4-079a2d0234b9-kube-api-access-gd76w\") pod \"maas-api-revoke-sub-d88717da-tkgzs\" (UID: \"496633ba-dc94-4a3c-a6c4-079a2d0234b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs"
Oct 30 23:28:34.098111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:34.098093    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496633ba-dc94-4a3c-a6c4-079a2d0234b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d88717da-tkgzs\" (UID: \"496633ba-dc94-4a3c-a6c4-079a2d0234b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs"
Oct 30 23:28:34.099881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:34.099851    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496633ba-dc94-4a3c-a6c4-079a2d0234b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d88717da-tkgzs\" (UID: \"496633ba-dc94-4a3c-a6c4-079a2d0234b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs"
Oct 30 23:28:34.154791 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:34.154770    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd76w\" (UniqueName: \"kubernetes.io/projected/496633ba-dc94-4a3c-a6c4-079a2d0234b9-kube-api-access-gd76w\") pod \"maas-api-revoke-sub-d88717da-tkgzs\" (UID: \"496633ba-dc94-4a3c-a6c4-079a2d0234b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs"
Oct 30 23:28:34.227005 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:34.226971    5078 generic.go:356] "Generic (PLEG): container finished" podID="8b0a87d8-687d-4ebf-bd66-aa6cc59701b0" containerID="5c146234b9514d86a5b35b61431b884d39ae0799fe6a3db786526b41fb7ca94d" exitCode=0
Oct 30 23:28:34.227154 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:34.227045    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd" event={"ID":"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0","Type":"ContainerDied","Data":"5c146234b9514d86a5b35b61431b884d39ae0799fe6a3db786526b41fb7ca94d"}
Oct 30 23:28:34.228525 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:34.228505    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf" event={"ID":"b8f824de-6ac1-491a-abc3-d4a941873bdf","Type":"ContainerStarted","Data":"bef2e68cfb57c5f2cc2e0eab8cb0039eeb5e4614efe418fb7a5032780e521bde"}
Oct 30 23:28:34.228659 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:34.228541    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf" event={"ID":"b8f824de-6ac1-491a-abc3-d4a941873bdf","Type":"ContainerStarted","Data":"a8bd3c9fb490f21a0605216f8914ab180a759bb352c5588b883db1856322e496"}
Oct 30 23:28:34.248031 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:34.248009    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs"
Oct 30 23:28:34.521872 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:34.521761    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf" podStartSLOduration=1.521745473 podStartE2EDuration="1.521745473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:28:34.521334491 +0000 UTC m=+1300.409240845" watchObservedRunningTime="2026-10-30 23:28:34.521745473 +0000 UTC m=+1300.409651826"
Oct 30 23:28:34.628295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:34.628274    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs"]
Oct 30 23:28:34.628390 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:28:34.628335    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496633ba_dc94_4a3c_a6c4_079a2d0234b9.slice/crio-849c9ad7498a729ccd86287167c7cc7786964400fff80fa6fc9c5e1dc04d6de9 WatchSource:0}: Error finding container 849c9ad7498a729ccd86287167c7cc7786964400fff80fa6fc9c5e1dc04d6de9: Status 404 returned error can't find the container with id 849c9ad7498a729ccd86287167c7cc7786964400fff80fa6fc9c5e1dc04d6de9
Oct 30 23:28:35.387622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:35.378548    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs" event={"ID":"496633ba-dc94-4a3c-a6c4-079a2d0234b9","Type":"ContainerStarted","Data":"7614ae45d3c00f9148e7699b8cddc68975d72397150717772c7229874b67ec7d"}
Oct 30 23:28:35.387622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:35.385673    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs" event={"ID":"496633ba-dc94-4a3c-a6c4-079a2d0234b9","Type":"ContainerStarted","Data":"849c9ad7498a729ccd86287167c7cc7786964400fff80fa6fc9c5e1dc04d6de9"}
Oct 30 23:28:35.412621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:35.388275    5078 generic.go:356] "Generic (PLEG): container finished" podID="b8f824de-6ac1-491a-abc3-d4a941873bdf" containerID="bef2e68cfb57c5f2cc2e0eab8cb0039eeb5e4614efe418fb7a5032780e521bde" exitCode=0
Oct 30 23:28:35.412621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:35.390795    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf" event={"ID":"b8f824de-6ac1-491a-abc3-d4a941873bdf","Type":"ContainerDied","Data":"bef2e68cfb57c5f2cc2e0eab8cb0039eeb5e4614efe418fb7a5032780e521bde"}
Oct 30 23:28:35.505638 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:35.502832    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs" podStartSLOduration=2.502814163 podStartE2EDuration="2.502814163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:28:35.501538914 +0000 UTC m=+1301.389445268" watchObservedRunningTime="2026-10-30 23:28:35.502814163 +0000 UTC m=+1301.390720518"
Oct 30 23:28:36.653642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:36.642863    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd"
Oct 30 23:28:36.653642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:36.643575    5078 generic.go:356] "Generic (PLEG): container finished" podID="496633ba-dc94-4a3c-a6c4-079a2d0234b9" containerID="7614ae45d3c00f9148e7699b8cddc68975d72397150717772c7229874b67ec7d" exitCode=0
Oct 30 23:28:36.653642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:36.649106    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs" event={"ID":"496633ba-dc94-4a3c-a6c4-079a2d0234b9","Type":"ContainerDied","Data":"7614ae45d3c00f9148e7699b8cddc68975d72397150717772c7229874b67ec7d"}
Oct 30 23:28:36.670645 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:36.668779    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:28:36.725673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:36.719024    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-maas-api-openshift-ca\") pod \"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0\" (UID: \"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0\") "
Oct 30 23:28:36.725673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:36.719144    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-maas-api-xks-ca\") pod \"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0\" (UID: \"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0\") "
Oct 30 23:28:36.725673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:36.719184    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-kube-api-access-hb9s8\" (UniqueName: \"kubernetes.io/projected/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-kube-api-access-hb9s8\") pod \"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0\" (UID: \"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0\") "
Oct 30 23:28:36.733623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:36.730789    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-maas-api-openshift-ca" pod "8b0a87d8-687d-4ebf-bd66-aa6cc59701b0" (UID: "8b0a87d8-687d-4ebf-bd66-aa6cc59701b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:28:36.779718 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:36.775773    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-maas-api-xks-ca" pod "8b0a87d8-687d-4ebf-bd66-aa6cc59701b0" (UID: "8b0a87d8-687d-4ebf-bd66-aa6cc59701b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:28:36.779718 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:36.778096    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-kube-api-access-hb9s8" pod "8b0a87d8-687d-4ebf-bd66-aa6cc59701b0" (UID: "8b0a87d8-687d-4ebf-bd66-aa6cc59701b0"). InnerVolumeSpecName "kube-api-access-hb9s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:28:36.831629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:36.829189    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:36.831629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:36.829234    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:36.831629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:36.829250    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb9s8\" (UniqueName: \"kubernetes.io/projected/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0-kube-api-access-hb9s8\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:37.312823 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.312778    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 23:28:37.433673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.426353    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:28:37.544621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.531578    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Oct 30 23:28:37.587007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.564198    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 23:28:37.667666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.574153    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:28:37.667666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.611881    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 30 23:28:37.667666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.621289    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:28:37.683644 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.683221    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:28:37.683843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.683694    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:28:37.700953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.688728    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:28:37.700953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.688790    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj" podUID="660a691a-733b-4ea7-a3db-4c55f649b099" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:28:37.737922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.737885    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf"
Oct 30 23:28:37.738322 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.738295    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd" event={"ID":"8b0a87d8-687d-4ebf-bd66-aa6cc59701b0","Type":"ContainerDied","Data":"0c6ad2168f20df2338381c769f701d65a5559e5786f715a99ec024faf49632ae"}
Oct 30 23:28:37.738512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.738496    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c6ad2168f20df2338381c769f701d65a5559e5786f715a99ec024faf49632ae"
Oct 30 23:28:37.745932 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.745908    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf" event={"ID":"b8f824de-6ac1-491a-abc3-d4a941873bdf","Type":"ContainerDied","Data":"a8bd3c9fb490f21a0605216f8914ab180a759bb352c5588b883db1856322e496"}
Oct 30 23:28:37.746067 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.746051    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8bd3c9fb490f21a0605216f8914ab180a759bb352c5588b883db1856322e496"
Oct 30 23:28:37.746159 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.741885    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf"
Oct 30 23:28:37.746428 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.739740    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd"
Oct 30 23:28:37.837465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.837418    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:28:37.875274 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.874400    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8f824de-6ac1-491a-abc3-d4a941873bdf-kube-api-access-tch42\" (UniqueName: \"kubernetes.io/projected/b8f824de-6ac1-491a-abc3-d4a941873bdf-kube-api-access-tch42\") pod \"b8f824de-6ac1-491a-abc3-d4a941873bdf\" (UID: \"b8f824de-6ac1-491a-abc3-d4a941873bdf\") "
Oct 30 23:28:37.875454 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.875348    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8f824de-6ac1-491a-abc3-d4a941873bdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8f824de-6ac1-491a-abc3-d4a941873bdf-maas-api-openshift-ca\") pod \"b8f824de-6ac1-491a-abc3-d4a941873bdf\" (UID: \"b8f824de-6ac1-491a-abc3-d4a941873bdf\") "
Oct 30 23:28:37.875454 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.875418    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8f824de-6ac1-491a-abc3-d4a941873bdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8f824de-6ac1-491a-abc3-d4a941873bdf-maas-api-xks-ca\") pod \"b8f824de-6ac1-491a-abc3-d4a941873bdf\" (UID: \"b8f824de-6ac1-491a-abc3-d4a941873bdf\") "
Oct 30 23:28:37.940694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.933634    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8f824de-6ac1-491a-abc3-d4a941873bdf-maas-api-openshift-ca" pod "b8f824de-6ac1-491a-abc3-d4a941873bdf" (UID: "b8f824de-6ac1-491a-abc3-d4a941873bdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:28:37.965628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.958900    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8f824de-6ac1-491a-abc3-d4a941873bdf-kube-api-access-tch42" pod "b8f824de-6ac1-491a-abc3-d4a941873bdf" (UID: "b8f824de-6ac1-491a-abc3-d4a941873bdf"). InnerVolumeSpecName "kube-api-access-tch42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:28:37.965628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.958989    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 23:28:37.965628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.959859    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8f824de-6ac1-491a-abc3-d4a941873bdf-maas-api-xks-ca" pod "b8f824de-6ac1-491a-abc3-d4a941873bdf" (UID: "b8f824de-6ac1-491a-abc3-d4a941873bdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:28:38.019629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:37.989461    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tch42\" (UniqueName: \"kubernetes.io/projected/b8f824de-6ac1-491a-abc3-d4a941873bdf-kube-api-access-tch42\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:38.042686 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.042658    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8f824de-6ac1-491a-abc3-d4a941873bdf-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:38.042934 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.042914    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8f824de-6ac1-491a-abc3-d4a941873bdf-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:38.095625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.084410    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:28:38.458623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.452830    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:28:38.545978 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.545917    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:28:38.546174 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.546003    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc" podUID="b964ddd5-6455-412c-85a8-07cd6b15cf40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:28:38.547185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.547019    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:28:38.547185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.547145    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j" podUID="3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:28:38.656480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.656436    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:28:38.659282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.656516    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs"
Oct 30 23:28:38.662416 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.662394    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:28:38.662636 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.662588    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:28:38.680199 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.680050    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/496633ba-dc94-4a3c-a6c4-079a2d0234b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496633ba-dc94-4a3c-a6c4-079a2d0234b9-maas-api-openshift-ca\") pod \"496633ba-dc94-4a3c-a6c4-079a2d0234b9\" (UID: \"496633ba-dc94-4a3c-a6c4-079a2d0234b9\") "
Oct 30 23:28:38.680199 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.680117    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496633ba-dc94-4a3c-a6c4-079a2d0234b9-kube-api-access-gd76w\" (UniqueName: \"kubernetes.io/projected/496633ba-dc94-4a3c-a6c4-079a2d0234b9-kube-api-access-gd76w\") pod \"496633ba-dc94-4a3c-a6c4-079a2d0234b9\" (UID: \"496633ba-dc94-4a3c-a6c4-079a2d0234b9\") "
Oct 30 23:28:38.680199 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.680153    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/496633ba-dc94-4a3c-a6c4-079a2d0234b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496633ba-dc94-4a3c-a6c4-079a2d0234b9-maas-api-xks-ca\") pod \"496633ba-dc94-4a3c-a6c4-079a2d0234b9\" (UID: \"496633ba-dc94-4a3c-a6c4-079a2d0234b9\") "
Oct 30 23:28:38.682191 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.681745    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496633ba-dc94-4a3c-a6c4-079a2d0234b9-maas-api-openshift-ca" pod "496633ba-dc94-4a3c-a6c4-079a2d0234b9" (UID: "496633ba-dc94-4a3c-a6c4-079a2d0234b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:28:38.683787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.683762    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/496633ba-dc94-4a3c-a6c4-079a2d0234b9-maas-api-xks-ca" pod "496633ba-dc94-4a3c-a6c4-079a2d0234b9" (UID: "496633ba-dc94-4a3c-a6c4-079a2d0234b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:28:38.684221 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.684194    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496633ba-dc94-4a3c-a6c4-079a2d0234b9-kube-api-access-gd76w" pod "496633ba-dc94-4a3c-a6c4-079a2d0234b9" (UID: "496633ba-dc94-4a3c-a6c4-079a2d0234b9"). InnerVolumeSpecName "kube-api-access-gd76w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:28:38.751483 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.751454    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs"
Oct 30 23:28:38.751483 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.751474    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs" event={"ID":"496633ba-dc94-4a3c-a6c4-079a2d0234b9","Type":"ContainerDied","Data":"849c9ad7498a729ccd86287167c7cc7786964400fff80fa6fc9c5e1dc04d6de9"}
Oct 30 23:28:38.751699 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.751516    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="849c9ad7498a729ccd86287167c7cc7786964400fff80fa6fc9c5e1dc04d6de9"
Oct 30 23:28:38.781582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.781553    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496633ba-dc94-4a3c-a6c4-079a2d0234b9-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:38.781821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.781800    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd76w\" (UniqueName: \"kubernetes.io/projected/496633ba-dc94-4a3c-a6c4-079a2d0234b9-kube-api-access-gd76w\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:38.781909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:38.781828    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496633ba-dc94-4a3c-a6c4-079a2d0234b9-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:41.049240 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:41.049210    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc"]
Oct 30 23:28:41.098980 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:41.098950    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f988c687-bbtvc"]
Oct 30 23:28:41.102412 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:41.102401    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs"]
Oct 30 23:28:41.105446 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:41.105425    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3ee03898-z8cfs"]
Oct 30 23:28:42.299967 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:42.299932    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3" path="/var/lib/kubelet/pods/3e4ccf4d-8f88-42d6-bc04-fdc7c5cfade3/volumes"
Oct 30 23:28:42.300435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:42.300421    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b5a4f8-65b5-4d66-8921-521ff24d61e1" path="/var/lib/kubelet/pods/d2b5a4f8-65b5-4d66-8921-521ff24d61e1/volumes"
Oct 30 23:28:43.228607 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.228561    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc"]
Oct 30 23:28:43.229116 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.229102    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b8f824de-6ac1-491a-abc3-d4a941873bdf" containerName="revoke-keys"
Oct 30 23:28:43.229162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.229117    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8f824de-6ac1-491a-abc3-d4a941873bdf" containerName="revoke-keys"
Oct 30 23:28:43.229162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.229142    5078 cpu_manager.go:414] "containerMap: removing container" podUID="8b0a87d8-687d-4ebf-bd66-aa6cc59701b0" containerName="revoke-keys"
Oct 30 23:28:43.229162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.229148    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b0a87d8-687d-4ebf-bd66-aa6cc59701b0" containerName="revoke-keys"
Oct 30 23:28:43.229252 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.229176    5078 cpu_manager.go:414] "containerMap: removing container" podUID="496633ba-dc94-4a3c-a6c4-079a2d0234b9" containerName="revoke-keys"
Oct 30 23:28:43.229252 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.229181    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="496633ba-dc94-4a3c-a6c4-079a2d0234b9" containerName="revoke-keys"
Oct 30 23:28:43.229306 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.229261    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b0a87d8-687d-4ebf-bd66-aa6cc59701b0" containerName="revoke-keys"
Oct 30 23:28:43.229306 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.229274    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="496633ba-dc94-4a3c-a6c4-079a2d0234b9" containerName="revoke-keys"
Oct 30 23:28:43.229306 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.229283    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8f824de-6ac1-491a-abc3-d4a941873bdf" containerName="revoke-keys"
Oct 30 23:28:43.233852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.233836    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc"
Oct 30 23:28:43.288815 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.288789    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc"]
Oct 30 23:28:43.334092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.334062    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:28:43.334486 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.334102    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:28:43.335902 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.335885    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:28:43.335902 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.335892    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:28:43.350069 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.350045    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb93fbfc-171e-44bc-867c-25e3cfdd883a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e1c4370-8z7kc\" (UID: \"fb93fbfc-171e-44bc-867c-25e3cfdd883a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc"
Oct 30 23:28:43.350176 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.350133    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb93fbfc-171e-44bc-867c-25e3cfdd883a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e1c4370-8z7kc\" (UID: \"fb93fbfc-171e-44bc-867c-25e3cfdd883a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc"
Oct 30 23:28:43.350292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.350271    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfwwc\" (UniqueName: \"kubernetes.io/projected/fb93fbfc-171e-44bc-867c-25e3cfdd883a-kube-api-access-tfwwc\") pod \"maas-api-revoke-sub-7e1c4370-8z7kc\" (UID: \"fb93fbfc-171e-44bc-867c-25e3cfdd883a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc"
Oct 30 23:28:43.423684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.423646    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"]
Oct 30 23:28:43.427760 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.427744    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.451181 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.451160    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"]
Oct 30 23:28:43.451763 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.451739    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d730be04-38d0-48a6-8235-20372487491f-istio-token\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.451816 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.451800    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-workload-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.451885 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.451864    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgg5q\" (UniqueName: \"kubernetes.io/projected/d730be04-38d0-48a6-8235-20372487491f-kube-api-access-rgg5q\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.451919 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.451908    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb93fbfc-171e-44bc-867c-25e3cfdd883a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e1c4370-8z7kc\" (UID: \"fb93fbfc-171e-44bc-867c-25e3cfdd883a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc"
Oct 30 23:28:43.452059 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.452045    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb93fbfc-171e-44bc-867c-25e3cfdd883a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e1c4370-8z7kc\" (UID: \"fb93fbfc-171e-44bc-867c-25e3cfdd883a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc"
Oct 30 23:28:43.452092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.452069    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-workload-certs\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.452136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.452101    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-istio-envoy\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.452136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.452122    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-istio-data\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.452297 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.452274    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d730be04-38d0-48a6-8235-20372487491f-istiod-ca-cert\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.452402 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.452384    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwwc\" (UniqueName: \"kubernetes.io/projected/fb93fbfc-171e-44bc-867c-25e3cfdd883a-kube-api-access-tfwwc\") pod \"maas-api-revoke-sub-7e1c4370-8z7kc\" (UID: \"fb93fbfc-171e-44bc-867c-25e3cfdd883a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc"
Oct 30 23:28:43.452489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.452474    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-credential-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.452556 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.452506    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d730be04-38d0-48a6-8235-20372487491f-istio-podinfo\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.452727 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.452707    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb93fbfc-171e-44bc-867c-25e3cfdd883a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e1c4370-8z7kc\" (UID: \"fb93fbfc-171e-44bc-867c-25e3cfdd883a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc"
Oct 30 23:28:43.454575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.454560    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb93fbfc-171e-44bc-867c-25e3cfdd883a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e1c4370-8z7kc\" (UID: \"fb93fbfc-171e-44bc-867c-25e3cfdd883a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc"
Oct 30 23:28:43.455458 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.455445    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.487143 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.487086    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"]
Oct 30 23:28:43.509211 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.509180    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwwc\" (UniqueName: \"kubernetes.io/projected/fb93fbfc-171e-44bc-867c-25e3cfdd883a-kube-api-access-tfwwc\") pod \"maas-api-revoke-sub-7e1c4370-8z7kc\" (UID: \"fb93fbfc-171e-44bc-867c-25e3cfdd883a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc"
Oct 30 23:28:43.546918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.546882    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"]
Oct 30 23:28:43.547204 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.547184    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc"
Oct 30 23:28:43.554132 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.554109    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-podinfo\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.554233 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.554146    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnwt6\" (UniqueName: \"kubernetes.io/projected/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-kube-api-access-mnwt6\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.554293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.554276    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-credential-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.554351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.554297    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istiod-ca-cert\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.554351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.554317    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d730be04-38d0-48a6-8235-20372487491f-istio-podinfo\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.554458 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.554422    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d730be04-38d0-48a6-8235-20372487491f-istio-token\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.554519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.554466    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-credential-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.554519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.554495    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-workload-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.554654 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.554531    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgg5q\" (UniqueName: \"kubernetes.io/projected/d730be04-38d0-48a6-8235-20372487491f-kube-api-access-rgg5q\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.554654 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.554580    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-token\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.554768 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.554687    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-credential-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.554820 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.554804    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-workload-certs\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.554888 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.554862    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-workload-certs\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.554941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.554908    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-envoy\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.554990 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.554942    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-istio-envoy\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.554990 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.554968    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-istio-data\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.554990 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.554972    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-workload-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.555212 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.555181    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-data\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.555286 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.555241    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d730be04-38d0-48a6-8235-20372487491f-istiod-ca-cert\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.555342 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.555316    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-workload-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.556036 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.555513    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-workload-certs\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.556036 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.555725    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-istio-data\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.556036 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.556007    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d730be04-38d0-48a6-8235-20372487491f-istiod-ca-cert\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.558827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.558807    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-istio-envoy\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.559156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.559135    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d730be04-38d0-48a6-8235-20372487491f-istio-podinfo\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.612577 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.612551    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d730be04-38d0-48a6-8235-20372487491f-istio-token\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.614252 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.614178    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgg5q\" (UniqueName: \"kubernetes.io/projected/d730be04-38d0-48a6-8235-20372487491f-kube-api-access-rgg5q\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.656612 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.656552    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-token\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.656779 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.656765    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-workload-certs\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.656849 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.656801    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-envoy\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.656917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.656852    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-data\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.656977 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.656930    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-workload-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.656977 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.656970    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-podinfo\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.657081 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.657005    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnwt6\" (UniqueName: \"kubernetes.io/projected/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-kube-api-access-mnwt6\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.657138 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.657076    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istiod-ca-cert\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.657138 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.657128    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-credential-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.657237 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.657197    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-workload-certs\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.658198 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.657538    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-credential-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.658198 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.657674    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-data\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.658198 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.657978    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-workload-socket\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.658198 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.658154    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istiod-ca-cert\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.659856 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.659832    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-envoy\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.660505 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.660483    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-podinfo\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.713724 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.713697    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-token\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.714173 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.714148    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnwt6\" (UniqueName: \"kubernetes.io/projected/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-kube-api-access-mnwt6\") pod \"e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") " pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.742168 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.742105    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:43.780513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.780489    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:43.913457 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.913425    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj"]
Oct 30 23:28:43.918847 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.918816    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj"
Oct 30 23:28:43.926766 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.926742    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px"]
Oct 30 23:28:43.949922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.949840    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px"
Oct 30 23:28:43.953066 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.953042    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc"]
Oct 30 23:28:43.960915 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.960893    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzw7p\" (UniqueName: \"kubernetes.io/projected/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-kube-api-access-rzw7p\") pod \"maas-api-revoke-sub-8785bf53-2r7mj\" (UID: \"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj"
Oct 30 23:28:43.961006 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.960942    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8785bf53-2r7mj\" (UID: \"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj"
Oct 30 23:28:43.961100 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.961080    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8785bf53-2r7mj\" (UID: \"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj"
Oct 30 23:28:43.973376 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:43.973353    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj"]
Oct 30 23:28:44.039323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.039258    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px"]
Oct 30 23:28:44.062769 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.062736    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzl4b\" (UniqueName: \"kubernetes.io/projected/6094da88-cad0-4909-81be-8bb07fb4154e-kube-api-access-nzl4b\") pod \"maas-api-revoke-sub-538ff61a-8h6px\" (UID: \"6094da88-cad0-4909-81be-8bb07fb4154e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px"
Oct 30 23:28:44.063022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.062986    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzw7p\" (UniqueName: \"kubernetes.io/projected/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-kube-api-access-rzw7p\") pod \"maas-api-revoke-sub-8785bf53-2r7mj\" (UID: \"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj"
Oct 30 23:28:44.063120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.063044    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8785bf53-2r7mj\" (UID: \"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj"
Oct 30 23:28:44.063200 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.063172    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6094da88-cad0-4909-81be-8bb07fb4154e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-538ff61a-8h6px\" (UID: \"6094da88-cad0-4909-81be-8bb07fb4154e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px"
Oct 30 23:28:44.063258 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.063222    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6094da88-cad0-4909-81be-8bb07fb4154e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-538ff61a-8h6px\" (UID: \"6094da88-cad0-4909-81be-8bb07fb4154e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px"
Oct 30 23:28:44.063319 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.063280    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8785bf53-2r7mj\" (UID: \"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj"
Oct 30 23:28:44.063982 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.063959    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8785bf53-2r7mj\" (UID: \"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj"
Oct 30 23:28:44.065937 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.065916    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8785bf53-2r7mj\" (UID: \"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj"
Oct 30 23:28:44.120415 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.120390    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzw7p\" (UniqueName: \"kubernetes.io/projected/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-kube-api-access-rzw7p\") pod \"maas-api-revoke-sub-8785bf53-2r7mj\" (UID: \"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj"
Oct 30 23:28:44.126516 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:28:44.126494    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd730be04_38d0_48a6_8235_20372487491f.slice/crio-21ffe91414717bf686c82aff297089d139ba65d25f1bfcf01d2e68bb9338f7fb WatchSource:0}: Error finding container 21ffe91414717bf686c82aff297089d139ba65d25f1bfcf01d2e68bb9338f7fb: Status 404 returned error can't find the container with id 21ffe91414717bf686c82aff297089d139ba65d25f1bfcf01d2e68bb9338f7fb
Oct 30 23:28:44.127272 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.127250    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"]
Oct 30 23:28:44.129377 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.129348    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:28:44.129465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.129408    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:28:44.129465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.129438    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:28:44.164570 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.164530    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6094da88-cad0-4909-81be-8bb07fb4154e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-538ff61a-8h6px\" (UID: \"6094da88-cad0-4909-81be-8bb07fb4154e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px"
Oct 30 23:28:44.164677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.164580    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6094da88-cad0-4909-81be-8bb07fb4154e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-538ff61a-8h6px\" (UID: \"6094da88-cad0-4909-81be-8bb07fb4154e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px"
Oct 30 23:28:44.164755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.164737    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzl4b\" (UniqueName: \"kubernetes.io/projected/6094da88-cad0-4909-81be-8bb07fb4154e-kube-api-access-nzl4b\") pod \"maas-api-revoke-sub-538ff61a-8h6px\" (UID: \"6094da88-cad0-4909-81be-8bb07fb4154e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px"
Oct 30 23:28:44.165347 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.165325    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6094da88-cad0-4909-81be-8bb07fb4154e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-538ff61a-8h6px\" (UID: \"6094da88-cad0-4909-81be-8bb07fb4154e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px"
Oct 30 23:28:44.167068 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.167052    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6094da88-cad0-4909-81be-8bb07fb4154e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-538ff61a-8h6px\" (UID: \"6094da88-cad0-4909-81be-8bb07fb4154e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px"
Oct 30 23:28:44.178007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.177982    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"]
Oct 30 23:28:44.178204 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:28:44.178182    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23a8a20_2deb_4b62_ba67_e1ae1d15463c.slice/crio-064882dd81de291286994a1f0a8d3a2b9bbc15ff3751b8d29d61fa421703909b WatchSource:0}: Error finding container 064882dd81de291286994a1f0a8d3a2b9bbc15ff3751b8d29d61fa421703909b: Status 404 returned error can't find the container with id 064882dd81de291286994a1f0a8d3a2b9bbc15ff3751b8d29d61fa421703909b
Oct 30 23:28:44.180897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.180864    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:28:44.181005 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.180928    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:28:44.181005 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.180956    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:28:44.221862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.221836    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzl4b\" (UniqueName: \"kubernetes.io/projected/6094da88-cad0-4909-81be-8bb07fb4154e-kube-api-access-nzl4b\") pod \"maas-api-revoke-sub-538ff61a-8h6px\" (UID: \"6094da88-cad0-4909-81be-8bb07fb4154e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px"
Oct 30 23:28:44.237372 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.237343    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj"
Oct 30 23:28:44.268615 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.265617    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px"
Oct 30 23:28:44.651641 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.649329    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj"]
Oct 30 23:28:44.651641 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:28:44.650266    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1deebc5_fa99_4e05_9ca2_6122a1fa20bc.slice/crio-5ec7344c6c2415f166963f9b765a3ce5c10c73592a17e580aa634b026f4d7c77 WatchSource:0}: Error finding container 5ec7344c6c2415f166963f9b765a3ce5c10c73592a17e580aa634b026f4d7c77: Status 404 returned error can't find the container with id 5ec7344c6c2415f166963f9b765a3ce5c10c73592a17e580aa634b026f4d7c77
Oct 30 23:28:44.712870 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:28:44.703714    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6094da88_cad0_4909_81be_8bb07fb4154e.slice/crio-6927eac614f8f6708420a01c557b4d015795de12e3822cf7cf1e2c2498791cc7 WatchSource:0}: Error finding container 6927eac614f8f6708420a01c557b4d015795de12e3822cf7cf1e2c2498791cc7: Status 404 returned error can't find the container with id 6927eac614f8f6708420a01c557b4d015795de12e3822cf7cf1e2c2498791cc7
Oct 30 23:28:44.712870 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.705663    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px"]
Oct 30 23:28:44.712870 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.711931    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"]
Oct 30 23:28:44.720971 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.720945    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.777911 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.777878    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1da554-65ce-4df0-a33b-761591100ac2-istio-token\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.778059 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.777942    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1da554-65ce-4df0-a33b-761591100ac2-istiod-ca-cert\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.778144 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.778060    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-istio-data\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.778144 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.778094    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-credential-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.778144 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.778130    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x68kc\" (UniqueName: \"kubernetes.io/projected/3a1da554-65ce-4df0-a33b-761591100ac2-kube-api-access-x68kc\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.778303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.778250    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-workload-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.778303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.778287    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-istio-envoy\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.778416 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.778310    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1da554-65ce-4df0-a33b-761591100ac2-istio-podinfo\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.778416 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.778349    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-workload-certs\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.783964 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.782634    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"]
Oct 30 23:28:44.806093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.806055    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px" event={"ID":"6094da88-cad0-4909-81be-8bb07fb4154e","Type":"ContainerStarted","Data":"6927eac614f8f6708420a01c557b4d015795de12e3822cf7cf1e2c2498791cc7"}
Oct 30 23:28:44.807962 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.807933    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc" event={"ID":"fb93fbfc-171e-44bc-867c-25e3cfdd883a","Type":"ContainerStarted","Data":"7732e8d5ff8dbcd2875e381e7f5b9239328d4fd97741496d325298b7717e1faa"}
Oct 30 23:28:44.808079 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.807989    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc" event={"ID":"fb93fbfc-171e-44bc-867c-25e3cfdd883a","Type":"ContainerStarted","Data":"e6bc31feddc30cddafa08b5ce171e0998b74bd8f4c84f2b0cd983f19c0818b54"}
Oct 30 23:28:44.809486 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.809459    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj" event={"ID":"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc","Type":"ContainerStarted","Data":"5ec7344c6c2415f166963f9b765a3ce5c10c73592a17e580aa634b026f4d7c77"}
Oct 30 23:28:44.811629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.811575    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf" event={"ID":"d23a8a20-2deb-4b62-ba67-e1ae1d15463c","Type":"ContainerStarted","Data":"42b714639ffd32286b83a316a61b57cda91bd5fa857d095d9a6dc528cc59722e"}
Oct 30 23:28:44.811724 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.811650    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf" event={"ID":"d23a8a20-2deb-4b62-ba67-e1ae1d15463c","Type":"ContainerStarted","Data":"064882dd81de291286994a1f0a8d3a2b9bbc15ff3751b8d29d61fa421703909b"}
Oct 30 23:28:44.813709 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.813691    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r" event={"ID":"d730be04-38d0-48a6-8235-20372487491f","Type":"ContainerStarted","Data":"dfe2a6b1451f0bec2f73294a0ef22e7253ef95e5f5aa083013f4f8478d778b38"}
Oct 30 23:28:44.813794 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.813728    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r" event={"ID":"d730be04-38d0-48a6-8235-20372487491f","Type":"ContainerStarted","Data":"21ffe91414717bf686c82aff297089d139ba65d25f1bfcf01d2e68bb9338f7fb"}
Oct 30 23:28:44.896822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.880372    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-workload-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.896822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.880430    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-istio-envoy\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.896822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.880456    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1da554-65ce-4df0-a33b-761591100ac2-istio-podinfo\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.896822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.880512    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-workload-certs\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.896822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.880576    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1da554-65ce-4df0-a33b-761591100ac2-istio-token\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.896822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.880694    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1da554-65ce-4df0-a33b-761591100ac2-istiod-ca-cert\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.896822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.880967    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-istio-data\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.896822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.881002    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-credential-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.896822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.881031    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x68kc\" (UniqueName: \"kubernetes.io/projected/3a1da554-65ce-4df0-a33b-761591100ac2-kube-api-access-x68kc\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.896822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.882029    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-workload-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.896822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.887082    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-workload-certs\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.896822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.887371    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-credential-socket\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.896822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.887392    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-istio-data\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.896822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.887970    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1da554-65ce-4df0-a33b-761591100ac2-istiod-ca-cert\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.972422 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.972343    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-istio-envoy\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.978265 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.972852    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r" podStartSLOduration=1.972833186 podStartE2EDuration="1.972833186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:28:44.97100812 +0000 UTC m=+1310.858914474" watchObservedRunningTime="2026-10-30 23:28:44.972833186 +0000 UTC m=+1310.860739540"
Oct 30 23:28:44.978265 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.977145    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1da554-65ce-4df0-a33b-761591100ac2-istio-token\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.978265 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.977756    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x68kc\" (UniqueName: \"kubernetes.io/projected/3a1da554-65ce-4df0-a33b-761591100ac2-kube-api-access-x68kc\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:44.978265 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:44.978217    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1da554-65ce-4df0-a33b-761591100ac2-istio-podinfo\") pod \"e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") " pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:45.109984 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:45.108303    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:45.114391 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:45.114331    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc" podStartSLOduration=2.114314175 podStartE2EDuration="2.114314175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:28:45.095063085 +0000 UTC m=+1310.982969441" watchObservedRunningTime="2026-10-30 23:28:45.114314175 +0000 UTC m=+1311.002220543"
Oct 30 23:28:45.218950 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:45.218850    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf" podStartSLOduration=2.2188339 podStartE2EDuration="2.2188339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:28:45.21657326 +0000 UTC m=+1311.104479614" watchObservedRunningTime="2026-10-30 23:28:45.2188339 +0000 UTC m=+1311.106740324"
Oct 30 23:28:45.564625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:45.557243    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"]
Oct 30 23:28:45.581115 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:28:45.581072    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1da554_65ce_4df0_a33b_761591100ac2.slice/crio-9de1ee282b2dac070b0da864ba4193494e418d2fd253aa7be4e259eb199bb3d7 WatchSource:0}: Error finding container 9de1ee282b2dac070b0da864ba4193494e418d2fd253aa7be4e259eb199bb3d7: Status 404 returned error can't find the container with id 9de1ee282b2dac070b0da864ba4193494e418d2fd253aa7be4e259eb199bb3d7
Oct 30 23:28:45.598968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:45.595976    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:28:45.598968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:45.596075    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:28:45.598968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:45.596124    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:28:45.759149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:45.744926    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:45.759149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:45.758727    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 23:28:45.759149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:45.758801    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r" podUID="d730be04-38d0-48a6-8235-20372487491f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 23:28:45.789621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:45.783319    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:45.789621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:45.783444    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 23:28:45.789621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:45.783489    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf" podUID="d23a8a20-2deb-4b62-ba67-e1ae1d15463c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 23:28:45.875876 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:45.856502    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px" event={"ID":"6094da88-cad0-4909-81be-8bb07fb4154e","Type":"ContainerStarted","Data":"67c92f7f4cd527d4a486f38041e1f9aaa44b64392972de3ae5318b20b469b3e0"}
Oct 30 23:28:45.938115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:45.937215    5078 generic.go:356] "Generic (PLEG): container finished" podID="fb93fbfc-171e-44bc-867c-25e3cfdd883a" containerID="7732e8d5ff8dbcd2875e381e7f5b9239328d4fd97741496d325298b7717e1faa" exitCode=0
Oct 30 23:28:45.938115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:45.937564    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc" event={"ID":"fb93fbfc-171e-44bc-867c-25e3cfdd883a","Type":"ContainerDied","Data":"7732e8d5ff8dbcd2875e381e7f5b9239328d4fd97741496d325298b7717e1faa"}
Oct 30 23:28:45.947940 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:45.947903    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj" event={"ID":"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc","Type":"ContainerStarted","Data":"963e4190b22bed3990e92e28c50e46cb39000035f9d6368821bcc2efc71cb10f"}
Oct 30 23:28:45.953968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:45.953936    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk" event={"ID":"3a1da554-65ce-4df0-a33b-761591100ac2","Type":"ContainerStarted","Data":"9de1ee282b2dac070b0da864ba4193494e418d2fd253aa7be4e259eb199bb3d7"}
Oct 30 23:28:46.019917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:46.011570    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px" podStartSLOduration=3.011550748 podStartE2EDuration="3.011550748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:28:45.983974213 +0000 UTC m=+1311.871880563" watchObservedRunningTime="2026-10-30 23:28:46.011550748 +0000 UTC m=+1311.899457105"
Oct 30 23:28:46.111470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:46.111442    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:46.114751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:46.114726    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 23:28:46.114948 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:46.114923    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk" podUID="3a1da554-65ce-4df0-a33b-761591100ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 23:28:46.228376 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:46.228326    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk" podStartSLOduration=2.22831059 podStartE2EDuration="2.22831059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:28:46.223960154 +0000 UTC m=+1312.111866511" watchObservedRunningTime="2026-10-30 23:28:46.22831059 +0000 UTC m=+1312.116216943"
Oct 30 23:28:46.341809 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:46.341758    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj" podStartSLOduration=3.341743019 podStartE2EDuration="3.341743019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:28:46.3377412 +0000 UTC m=+1312.225647578" watchObservedRunningTime="2026-10-30 23:28:46.341743019 +0000 UTC m=+1312.229649373"
Oct 30 23:28:46.743241 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:46.743142    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 23:28:46.743241 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:46.743205    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r" podUID="d730be04-38d0-48a6-8235-20372487491f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 23:28:46.793622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:46.782986    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 23:28:46.793622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:46.783047    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf" podUID="d23a8a20-2deb-4b62-ba67-e1ae1d15463c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 23:28:47.018095 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.017303    5078 generic.go:356] "Generic (PLEG): container finished" podID="6094da88-cad0-4909-81be-8bb07fb4154e" containerID="67c92f7f4cd527d4a486f38041e1f9aaa44b64392972de3ae5318b20b469b3e0" exitCode=0
Oct 30 23:28:47.018095 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.017494    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px" event={"ID":"6094da88-cad0-4909-81be-8bb07fb4154e","Type":"ContainerDied","Data":"67c92f7f4cd527d4a486f38041e1f9aaa44b64392972de3ae5318b20b469b3e0"}
Oct 30 23:28:47.020774 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.020448    5078 generic.go:356] "Generic (PLEG): container finished" podID="d1deebc5-fa99-4e05-9ca2-6122a1fa20bc" containerID="963e4190b22bed3990e92e28c50e46cb39000035f9d6368821bcc2efc71cb10f" exitCode=0
Oct 30 23:28:47.020774 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.020570    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj" event={"ID":"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc","Type":"ContainerDied","Data":"963e4190b22bed3990e92e28c50e46cb39000035f9d6368821bcc2efc71cb10f"}
Oct 30 23:28:47.033565 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.033532    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk" event={"ID":"3a1da554-65ce-4df0-a33b-761591100ac2","Type":"ContainerStarted","Data":"f4b4c3290c428cfe8fa298bf23f342aaa6895d5940cc203f87f2ff02a4cfde55"}
Oct 30 23:28:47.132702 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.131134    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 23:28:47.132702 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.131191    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk" podUID="3a1da554-65ce-4df0-a33b-761591100ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 23:28:47.655496 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.655446    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc"
Oct 30 23:28:47.730630 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.724753    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb93fbfc-171e-44bc-867c-25e3cfdd883a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb93fbfc-171e-44bc-867c-25e3cfdd883a-maas-api-xks-ca\") pod \"fb93fbfc-171e-44bc-867c-25e3cfdd883a\" (UID: \"fb93fbfc-171e-44bc-867c-25e3cfdd883a\") "
Oct 30 23:28:47.730630 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.724880    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb93fbfc-171e-44bc-867c-25e3cfdd883a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb93fbfc-171e-44bc-867c-25e3cfdd883a-maas-api-openshift-ca\") pod \"fb93fbfc-171e-44bc-867c-25e3cfdd883a\" (UID: \"fb93fbfc-171e-44bc-867c-25e3cfdd883a\") "
Oct 30 23:28:47.730630 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.724982    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb93fbfc-171e-44bc-867c-25e3cfdd883a-kube-api-access-tfwwc\" (UniqueName: \"kubernetes.io/projected/fb93fbfc-171e-44bc-867c-25e3cfdd883a-kube-api-access-tfwwc\") pod \"fb93fbfc-171e-44bc-867c-25e3cfdd883a\" (UID: \"fb93fbfc-171e-44bc-867c-25e3cfdd883a\") "
Oct 30 23:28:47.744621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.731778    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb93fbfc-171e-44bc-867c-25e3cfdd883a-maas-api-openshift-ca" pod "fb93fbfc-171e-44bc-867c-25e3cfdd883a" (UID: "fb93fbfc-171e-44bc-867c-25e3cfdd883a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:28:47.744621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.741301    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb93fbfc-171e-44bc-867c-25e3cfdd883a-maas-api-xks-ca" pod "fb93fbfc-171e-44bc-867c-25e3cfdd883a" (UID: "fb93fbfc-171e-44bc-867c-25e3cfdd883a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:28:47.744621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.741826    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb93fbfc-171e-44bc-867c-25e3cfdd883a-kube-api-access-tfwwc" pod "fb93fbfc-171e-44bc-867c-25e3cfdd883a" (UID: "fb93fbfc-171e-44bc-867c-25e3cfdd883a"). InnerVolumeSpecName "kube-api-access-tfwwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:28:47.744621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.743880    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 23:28:47.744621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.743934    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r" podUID="d730be04-38d0-48a6-8235-20372487491f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 23:28:47.748685 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:28:47.747946    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:28:47.792672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.782974    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 23:28:47.792672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.783031    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf" podUID="d23a8a20-2deb-4b62-ba67-e1ae1d15463c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 23:28:47.841672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.833770    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfwwc\" (UniqueName: \"kubernetes.io/projected/fb93fbfc-171e-44bc-867c-25e3cfdd883a-kube-api-access-tfwwc\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:47.841672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.833807    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb93fbfc-171e-44bc-867c-25e3cfdd883a-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:47.841672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:47.833824    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb93fbfc-171e-44bc-867c-25e3cfdd883a-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:48.049989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:48.049915    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc"
Oct 30 23:28:48.050234 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:48.050205    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc" event={"ID":"fb93fbfc-171e-44bc-867c-25e3cfdd883a","Type":"ContainerDied","Data":"e6bc31feddc30cddafa08b5ce171e0998b74bd8f4c84f2b0cd983f19c0818b54"}
Oct 30 23:28:48.050881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:48.050859    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6bc31feddc30cddafa08b5ce171e0998b74bd8f4c84f2b0cd983f19c0818b54"
Oct 30 23:28:48.117661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:48.117624    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 23:28:48.125623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:48.121677    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk" podUID="3a1da554-65ce-4df0-a33b-761591100ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 23:28:48.753414 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:48.753083    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 23:28:48.753414 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:48.753137    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r" podUID="d730be04-38d0-48a6-8235-20372487491f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 23:28:48.764626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:48.760769    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px"
Oct 30 23:28:48.794622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:48.789782    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 23:28:48.794622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:48.789839    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf" podUID="d23a8a20-2deb-4b62-ba67-e1ae1d15463c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 23:28:48.837790 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:48.837752    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj"
Oct 30 23:28:48.886623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:48.886144    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6094da88-cad0-4909-81be-8bb07fb4154e-kube-api-access-nzl4b\" (UniqueName: \"kubernetes.io/projected/6094da88-cad0-4909-81be-8bb07fb4154e-kube-api-access-nzl4b\") pod \"6094da88-cad0-4909-81be-8bb07fb4154e\" (UID: \"6094da88-cad0-4909-81be-8bb07fb4154e\") "
Oct 30 23:28:48.886623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:48.886254    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6094da88-cad0-4909-81be-8bb07fb4154e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6094da88-cad0-4909-81be-8bb07fb4154e-maas-api-xks-ca\") pod \"6094da88-cad0-4909-81be-8bb07fb4154e\" (UID: \"6094da88-cad0-4909-81be-8bb07fb4154e\") "
Oct 30 23:28:48.886623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:48.886427    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6094da88-cad0-4909-81be-8bb07fb4154e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6094da88-cad0-4909-81be-8bb07fb4154e-maas-api-openshift-ca\") pod \"6094da88-cad0-4909-81be-8bb07fb4154e\" (UID: \"6094da88-cad0-4909-81be-8bb07fb4154e\") "
Oct 30 23:28:48.900854 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:48.898989    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6094da88-cad0-4909-81be-8bb07fb4154e-maas-api-openshift-ca" pod "6094da88-cad0-4909-81be-8bb07fb4154e" (UID: "6094da88-cad0-4909-81be-8bb07fb4154e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:28:48.945565 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:48.926085    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6094da88-cad0-4909-81be-8bb07fb4154e-maas-api-xks-ca" pod "6094da88-cad0-4909-81be-8bb07fb4154e" (UID: "6094da88-cad0-4909-81be-8bb07fb4154e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:28:48.946078 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:48.946048    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6094da88-cad0-4909-81be-8bb07fb4154e-kube-api-access-nzl4b" pod "6094da88-cad0-4909-81be-8bb07fb4154e" (UID: "6094da88-cad0-4909-81be-8bb07fb4154e"). InnerVolumeSpecName "kube-api-access-nzl4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:28:49.007632 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.007058    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-maas-api-openshift-ca\") pod \"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc\" (UID: \"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc\") "
Oct 30 23:28:49.007632 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.007198    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-maas-api-xks-ca\") pod \"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc\" (UID: \"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc\") "
Oct 30 23:28:49.007632 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.007342    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-kube-api-access-rzw7p\" (UniqueName: \"kubernetes.io/projected/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-kube-api-access-rzw7p\") pod \"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc\" (UID: \"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc\") "
Oct 30 23:28:49.008771 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.008347    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzl4b\" (UniqueName: \"kubernetes.io/projected/6094da88-cad0-4909-81be-8bb07fb4154e-kube-api-access-nzl4b\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:49.008771 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.008379    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6094da88-cad0-4909-81be-8bb07fb4154e-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:49.008771 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.008395    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6094da88-cad0-4909-81be-8bb07fb4154e-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:49.018828 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.018794    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-maas-api-openshift-ca" pod "d1deebc5-fa99-4e05-9ca2-6122a1fa20bc" (UID: "d1deebc5-fa99-4e05-9ca2-6122a1fa20bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:28:49.041718 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.041682    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-maas-api-xks-ca" pod "d1deebc5-fa99-4e05-9ca2-6122a1fa20bc" (UID: "d1deebc5-fa99-4e05-9ca2-6122a1fa20bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:28:49.044817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.044771    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-kube-api-access-rzw7p" pod "d1deebc5-fa99-4e05-9ca2-6122a1fa20bc" (UID: "d1deebc5-fa99-4e05-9ca2-6122a1fa20bc"). InnerVolumeSpecName "kube-api-access-rzw7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:28:49.054881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.054828    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8"]
Oct 30 23:28:49.134624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.122694    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 23:28:49.136626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.134873    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk" podUID="3a1da554-65ce-4df0-a33b-761591100ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 23:28:49.136626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.135788    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj" event={"ID":"d1deebc5-fa99-4e05-9ca2-6122a1fa20bc","Type":"ContainerDied","Data":"5ec7344c6c2415f166963f9b765a3ce5c10c73592a17e580aa634b026f4d7c77"}
Oct 30 23:28:49.136626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.135845    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ec7344c6c2415f166963f9b765a3ce5c10c73592a17e580aa634b026f4d7c77"
Oct 30 23:28:49.136626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.135927    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj"
Oct 30 23:28:49.138292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.137885    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:49.138292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.137927    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:49.138292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.137943    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzw7p\" (UniqueName: \"kubernetes.io/projected/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc-kube-api-access-rzw7p\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:49.144127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.141681    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px"
Oct 30 23:28:49.144127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.143033    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px" event={"ID":"6094da88-cad0-4909-81be-8bb07fb4154e","Type":"ContainerDied","Data":"6927eac614f8f6708420a01c557b4d015795de12e3822cf7cf1e2c2498791cc7"}
Oct 30 23:28:49.144127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.143084    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6927eac614f8f6708420a01c557b4d015795de12e3822cf7cf1e2c2498791cc7"
Oct 30 23:28:49.144127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.143106    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9b87a6a-n5pt8"]
Oct 30 23:28:49.239899 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.239466    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5"]
Oct 30 23:28:49.240350 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.240328    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d1deebc5-fa99-4e05-9ca2-6122a1fa20bc" containerName="revoke-keys"
Oct 30 23:28:49.240477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.240352    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1deebc5-fa99-4e05-9ca2-6122a1fa20bc" containerName="revoke-keys"
Oct 30 23:28:49.240477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.240369    5078 cpu_manager.go:414] "containerMap: removing container" podUID="fb93fbfc-171e-44bc-867c-25e3cfdd883a" containerName="revoke-keys"
Oct 30 23:28:49.240477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.240376    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb93fbfc-171e-44bc-867c-25e3cfdd883a" containerName="revoke-keys"
Oct 30 23:28:49.240477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.240411    5078 cpu_manager.go:414] "containerMap: removing container" podUID="6094da88-cad0-4909-81be-8bb07fb4154e" containerName="revoke-keys"
Oct 30 23:28:49.240477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.240419    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6094da88-cad0-4909-81be-8bb07fb4154e" containerName="revoke-keys"
Oct 30 23:28:49.240816 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.240578    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1deebc5-fa99-4e05-9ca2-6122a1fa20bc" containerName="revoke-keys"
Oct 30 23:28:49.240816 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.240611    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb93fbfc-171e-44bc-867c-25e3cfdd883a" containerName="revoke-keys"
Oct 30 23:28:49.240816 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.240624    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="6094da88-cad0-4909-81be-8bb07fb4154e" containerName="revoke-keys"
Oct 30 23:28:49.244356 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.244336    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5"
Oct 30 23:28:49.343757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.343492    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5"]
Oct 30 23:28:49.343961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.343849    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:28:49.343961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.343885    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:28:49.343961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.343928    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:28:49.344134 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.344118    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:28:49.444227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.443911    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a09de6-56f1-4729-899d-3a0929e954f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2371acca-r5mb5\" (UID: \"b2a09de6-56f1-4729-899d-3a0929e954f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5"
Oct 30 23:28:49.444401 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.444356    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vk2l\" (UniqueName: \"kubernetes.io/projected/b2a09de6-56f1-4729-899d-3a0929e954f4-kube-api-access-2vk2l\") pod \"maas-api-revoke-sub-2371acca-r5mb5\" (UID: \"b2a09de6-56f1-4729-899d-3a0929e954f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5"
Oct 30 23:28:49.444445 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.444391    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a09de6-56f1-4729-899d-3a0929e954f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2371acca-r5mb5\" (UID: \"b2a09de6-56f1-4729-899d-3a0929e954f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5"
Oct 30 23:28:49.630033 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.582431    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a09de6-56f1-4729-899d-3a0929e954f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2371acca-r5mb5\" (UID: \"b2a09de6-56f1-4729-899d-3a0929e954f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5"
Oct 30 23:28:49.630033 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.582497    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vk2l\" (UniqueName: \"kubernetes.io/projected/b2a09de6-56f1-4729-899d-3a0929e954f4-kube-api-access-2vk2l\") pod \"maas-api-revoke-sub-2371acca-r5mb5\" (UID: \"b2a09de6-56f1-4729-899d-3a0929e954f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5"
Oct 30 23:28:49.630033 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.582657    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a09de6-56f1-4729-899d-3a0929e954f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2371acca-r5mb5\" (UID: \"b2a09de6-56f1-4729-899d-3a0929e954f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5"
Oct 30 23:28:49.630033 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.584560    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a09de6-56f1-4729-899d-3a0929e954f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2371acca-r5mb5\" (UID: \"b2a09de6-56f1-4729-899d-3a0929e954f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5"
Oct 30 23:28:49.630033 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.593020    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a09de6-56f1-4729-899d-3a0929e954f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2371acca-r5mb5\" (UID: \"b2a09de6-56f1-4729-899d-3a0929e954f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5"
Oct 30 23:28:49.732622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.720279    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vk2l\" (UniqueName: \"kubernetes.io/projected/b2a09de6-56f1-4729-899d-3a0929e954f4-kube-api-access-2vk2l\") pod \"maas-api-revoke-sub-2371acca-r5mb5\" (UID: \"b2a09de6-56f1-4729-899d-3a0929e954f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5"
Oct 30 23:28:49.742759 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.742727    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 23:28:49.742919 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.742793    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r" podUID="d730be04-38d0-48a6-8235-20372487491f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 23:28:49.784621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.783955    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 23:28:49.784621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.784026    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf" podUID="d23a8a20-2deb-4b62-ba67-e1ae1d15463c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 23:28:49.881674 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:49.880795    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5"
Oct 30 23:28:50.141724 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:50.137702    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 23:28:50.141724 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:50.137766    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk" podUID="3a1da554-65ce-4df0-a33b-761591100ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 23:28:50.350225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:50.350194    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc745fc-5bb4-4a2d-af49-02c5da152d46" path="/var/lib/kubelet/pods/3bc745fc-5bb4-4a2d-af49-02c5da152d46/volumes"
Oct 30 23:28:50.437734 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:28:50.437708    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a09de6_56f1_4729_899d_3a0929e954f4.slice/crio-1a80bd7fa4237278ff755755a71c38ad2f6c1fadf9c72a0d4408d01a485822cc WatchSource:0}: Error finding container 1a80bd7fa4237278ff755755a71c38ad2f6c1fadf9c72a0d4408d01a485822cc: Status 404 returned error can't find the container with id 1a80bd7fa4237278ff755755a71c38ad2f6c1fadf9c72a0d4408d01a485822cc
Oct 30 23:28:50.438030 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:50.438001    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5"]
Oct 30 23:28:50.742575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:50.742479    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body=
Oct 30 23:28:50.742575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:50.742547    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r" podUID="d730be04-38d0-48a6-8235-20372487491f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused"
Oct 30 23:28:50.789349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:50.782324    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Oct 30 23:28:50.789349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:50.782391    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf" podUID="d23a8a20-2deb-4b62-ba67-e1ae1d15463c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Oct 30 23:28:51.110904 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:51.110857    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 23:28:51.111106 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:51.110916    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk" podUID="3a1da554-65ce-4df0-a33b-761591100ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 23:28:51.195906 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:51.195868    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5" event={"ID":"b2a09de6-56f1-4729-899d-3a0929e954f4","Type":"ContainerStarted","Data":"8899f2e38b7515202d2f62a76a84f78db0aa5958085b812e066b4402ef6407d8"}
Oct 30 23:28:51.196416 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:51.195923    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5" event={"ID":"b2a09de6-56f1-4729-899d-3a0929e954f4","Type":"ContainerStarted","Data":"1a80bd7fa4237278ff755755a71c38ad2f6c1fadf9c72a0d4408d01a485822cc"}
Oct 30 23:28:51.334047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:51.333979    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5" podStartSLOduration=2.333963234 podStartE2EDuration="2.333963234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:28:51.329297627 +0000 UTC m=+1317.217203981" watchObservedRunningTime="2026-10-30 23:28:51.333963234 +0000 UTC m=+1317.221869586"
Oct 30 23:28:52.110948 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:52.110906    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 23:28:52.111164 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:52.110971    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk" podUID="3a1da554-65ce-4df0-a33b-761591100ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 23:28:52.207223 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:52.207185    5078 generic.go:356] "Generic (PLEG): container finished" podID="b2a09de6-56f1-4729-899d-3a0929e954f4" containerID="8899f2e38b7515202d2f62a76a84f78db0aa5958085b812e066b4402ef6407d8" exitCode=0
Oct 30 23:28:52.207687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:52.207314    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5" event={"ID":"b2a09de6-56f1-4729-899d-3a0929e954f4","Type":"ContainerDied","Data":"8899f2e38b7515202d2f62a76a84f78db0aa5958085b812e066b4402ef6407d8"}
Oct 30 23:28:52.267901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:52.267868    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:52.269240 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:52.269199    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:52.270047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:52.270022    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:28:52.352983 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:52.352929    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:52.353841 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:52.353815    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:52.355004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:52.354976    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:28:53.111373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:53.111342    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Oct 30 23:28:53.111542 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:53.111398    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk" podUID="3a1da554-65ce-4df0-a33b-761591100ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Oct 30 23:28:53.401163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:53.401138    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5"
Oct 30 23:28:53.453559 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:53.453529    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2a09de6-56f1-4729-899d-3a0929e954f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a09de6-56f1-4729-899d-3a0929e954f4-maas-api-xks-ca\") pod \"b2a09de6-56f1-4729-899d-3a0929e954f4\" (UID: \"b2a09de6-56f1-4729-899d-3a0929e954f4\") "
Oct 30 23:28:53.453772 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:53.453668    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2a09de6-56f1-4729-899d-3a0929e954f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a09de6-56f1-4729-899d-3a0929e954f4-maas-api-openshift-ca\") pod \"b2a09de6-56f1-4729-899d-3a0929e954f4\" (UID: \"b2a09de6-56f1-4729-899d-3a0929e954f4\") "
Oct 30 23:28:53.454128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:53.453861    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a09de6-56f1-4729-899d-3a0929e954f4-kube-api-access-2vk2l\" (UniqueName: \"kubernetes.io/projected/b2a09de6-56f1-4729-899d-3a0929e954f4-kube-api-access-2vk2l\") pod \"b2a09de6-56f1-4729-899d-3a0929e954f4\" (UID: \"b2a09de6-56f1-4729-899d-3a0929e954f4\") "
Oct 30 23:28:53.454128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:53.454087    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a09de6-56f1-4729-899d-3a0929e954f4-maas-api-openshift-ca" pod "b2a09de6-56f1-4729-899d-3a0929e954f4" (UID: "b2a09de6-56f1-4729-899d-3a0929e954f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:28:53.454467 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:53.454331    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2a09de6-56f1-4729-899d-3a0929e954f4-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:53.455910 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:53.455884    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a09de6-56f1-4729-899d-3a0929e954f4-maas-api-xks-ca" pod "b2a09de6-56f1-4729-899d-3a0929e954f4" (UID: "b2a09de6-56f1-4729-899d-3a0929e954f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:28:53.456091 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:53.456069    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a09de6-56f1-4729-899d-3a0929e954f4-kube-api-access-2vk2l" pod "b2a09de6-56f1-4729-899d-3a0929e954f4" (UID: "b2a09de6-56f1-4729-899d-3a0929e954f4"). InnerVolumeSpecName "kube-api-access-2vk2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:28:53.556278 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:53.556245    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vk2l\" (UniqueName: \"kubernetes.io/projected/b2a09de6-56f1-4729-899d-3a0929e954f4-kube-api-access-2vk2l\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:53.556278 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:53.556274    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2a09de6-56f1-4729-899d-3a0929e954f4-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:54.080297 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.080268    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2"]
Oct 30 23:28:54.080847 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.080831    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b2a09de6-56f1-4729-899d-3a0929e954f4" containerName="revoke-keys"
Oct 30 23:28:54.080847 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.080847    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a09de6-56f1-4729-899d-3a0929e954f4" containerName="revoke-keys"
Oct 30 23:28:54.080945 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.080938    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a09de6-56f1-4729-899d-3a0929e954f4" containerName="revoke-keys"
Oct 30 23:28:54.084785 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.084767    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2"
Oct 30 23:28:54.116376 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.116355    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:54.116652 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.116634    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:54.117335 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.117315    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:28:54.141097 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.141059    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2"]
Oct 30 23:28:54.163976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.163939    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2cc5\" (UniqueName: \"kubernetes.io/projected/c9e5057f-96ec-4a37-a54c-6c15c320283b-kube-api-access-v2cc5\") pod \"maas-api-revoke-sub-194230bb-5ndw2\" (UID: \"c9e5057f-96ec-4a37-a54c-6c15c320283b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2"
Oct 30 23:28:54.164121 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.164020    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e5057f-96ec-4a37-a54c-6c15c320283b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-194230bb-5ndw2\" (UID: \"c9e5057f-96ec-4a37-a54c-6c15c320283b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2"
Oct 30 23:28:54.164121 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.164051    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e5057f-96ec-4a37-a54c-6c15c320283b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-194230bb-5ndw2\" (UID: \"c9e5057f-96ec-4a37-a54c-6c15c320283b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2"
Oct 30 23:28:54.236548 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.236523    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5"
Oct 30 23:28:54.236702 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.236564    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5" event={"ID":"b2a09de6-56f1-4729-899d-3a0929e954f4","Type":"ContainerDied","Data":"1a80bd7fa4237278ff755755a71c38ad2f6c1fadf9c72a0d4408d01a485822cc"}
Oct 30 23:28:54.236702 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.236620    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a80bd7fa4237278ff755755a71c38ad2f6c1fadf9c72a0d4408d01a485822cc"
Oct 30 23:28:54.265162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.265134    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2cc5\" (UniqueName: \"kubernetes.io/projected/c9e5057f-96ec-4a37-a54c-6c15c320283b-kube-api-access-v2cc5\") pod \"maas-api-revoke-sub-194230bb-5ndw2\" (UID: \"c9e5057f-96ec-4a37-a54c-6c15c320283b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2"
Oct 30 23:28:54.265307 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.265213    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e5057f-96ec-4a37-a54c-6c15c320283b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-194230bb-5ndw2\" (UID: \"c9e5057f-96ec-4a37-a54c-6c15c320283b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2"
Oct 30 23:28:54.265307 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.265240    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e5057f-96ec-4a37-a54c-6c15c320283b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-194230bb-5ndw2\" (UID: \"c9e5057f-96ec-4a37-a54c-6c15c320283b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2"
Oct 30 23:28:54.265956 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.265932    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e5057f-96ec-4a37-a54c-6c15c320283b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-194230bb-5ndw2\" (UID: \"c9e5057f-96ec-4a37-a54c-6c15c320283b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2"
Oct 30 23:28:54.267849 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.267830    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e5057f-96ec-4a37-a54c-6c15c320283b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-194230bb-5ndw2\" (UID: \"c9e5057f-96ec-4a37-a54c-6c15c320283b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2"
Oct 30 23:28:54.324208 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.324179    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2cc5\" (UniqueName: \"kubernetes.io/projected/c9e5057f-96ec-4a37-a54c-6c15c320283b-kube-api-access-v2cc5\") pod \"maas-api-revoke-sub-194230bb-5ndw2\" (UID: \"c9e5057f-96ec-4a37-a54c-6c15c320283b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2"
Oct 30 23:28:54.400318 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.400254    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2"
Oct 30 23:28:54.553713 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.553682    5078 scope.go:122] "RemoveContainer" containerID="41de6460f6b68ce2be6701f5016721b499c97aa90c7b72e5e32e147fd8909f0e"
Oct 30 23:28:54.564339 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.564312    5078 scope.go:122] "RemoveContainer" containerID="e243e07ee5d344d5b86901584b272c3fed5d72c78f06b3492bc9e3fc71033ff2"
Oct 30 23:28:54.591475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.591447    5078 scope.go:122] "RemoveContainer" containerID="3158863374b626e91027b43deddf58154fbe9e6a8c8a68f6cf2fb56a1b2492ec"
Oct 30 23:28:54.797068 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:54.797042    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2"]
Oct 30 23:28:54.797068 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:28:54.797058    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e5057f_96ec_4a37_a54c_6c15c320283b.slice/crio-6db2291f0c1e7aba9d0c0c716e641e41bde185282fc33d927af8bbb3eec8f0c8 WatchSource:0}: Error finding container 6db2291f0c1e7aba9d0c0c716e641e41bde185282fc33d927af8bbb3eec8f0c8: Status 404 returned error can't find the container with id 6db2291f0c1e7aba9d0c0c716e641e41bde185282fc33d927af8bbb3eec8f0c8
Oct 30 23:28:55.246416 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:55.246370    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2" event={"ID":"c9e5057f-96ec-4a37-a54c-6c15c320283b","Type":"ContainerStarted","Data":"3741dad639941b454b9573c1a0f43ac5afd3f226e900eff303f656f16b0fa9d4"}
Oct 30 23:28:55.246584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:55.246471    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2" event={"ID":"c9e5057f-96ec-4a37-a54c-6c15c320283b","Type":"ContainerStarted","Data":"6db2291f0c1e7aba9d0c0c716e641e41bde185282fc33d927af8bbb3eec8f0c8"}
Oct 30 23:28:55.361216 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:55.361125    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2" podStartSLOduration=1.361111564 podStartE2EDuration="1.361111564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:28:55.360087625 +0000 UTC m=+1321.247993978" watchObservedRunningTime="2026-10-30 23:28:55.361111564 +0000 UTC m=+1321.249017917"
Oct 30 23:28:56.256478 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:56.256440    5078 generic.go:356] "Generic (PLEG): container finished" podID="c9e5057f-96ec-4a37-a54c-6c15c320283b" containerID="3741dad639941b454b9573c1a0f43ac5afd3f226e900eff303f656f16b0fa9d4" exitCode=0
Oct 30 23:28:56.256934 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:56.256518    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2" event={"ID":"c9e5057f-96ec-4a37-a54c-6c15c320283b","Type":"ContainerDied","Data":"3741dad639941b454b9573c1a0f43ac5afd3f226e900eff303f656f16b0fa9d4"}
Oct 30 23:28:57.052252 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:57.052160    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv"]
Oct 30 23:28:57.056922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:57.056883    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz"]
Oct 30 23:28:57.101677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:57.101643    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ecf73501-gfxkv"]
Oct 30 23:28:57.105694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:57.105669    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8a7d52ee-clhvz"]
Oct 30 23:28:57.451696 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:57.451672    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2"
Oct 30 23:28:57.530912 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:57.530888    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9e5057f-96ec-4a37-a54c-6c15c320283b-kube-api-access-v2cc5\" (UniqueName: \"kubernetes.io/projected/c9e5057f-96ec-4a37-a54c-6c15c320283b-kube-api-access-v2cc5\") pod \"c9e5057f-96ec-4a37-a54c-6c15c320283b\" (UID: \"c9e5057f-96ec-4a37-a54c-6c15c320283b\") "
Oct 30 23:28:57.531066 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:57.531048    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9e5057f-96ec-4a37-a54c-6c15c320283b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e5057f-96ec-4a37-a54c-6c15c320283b-maas-api-xks-ca\") pod \"c9e5057f-96ec-4a37-a54c-6c15c320283b\" (UID: \"c9e5057f-96ec-4a37-a54c-6c15c320283b\") "
Oct 30 23:28:57.531132 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:57.531095    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9e5057f-96ec-4a37-a54c-6c15c320283b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e5057f-96ec-4a37-a54c-6c15c320283b-maas-api-openshift-ca\") pod \"c9e5057f-96ec-4a37-a54c-6c15c320283b\" (UID: \"c9e5057f-96ec-4a37-a54c-6c15c320283b\") "
Oct 30 23:28:57.531445 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:57.531423    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9e5057f-96ec-4a37-a54c-6c15c320283b-maas-api-openshift-ca" pod "c9e5057f-96ec-4a37-a54c-6c15c320283b" (UID: "c9e5057f-96ec-4a37-a54c-6c15c320283b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:28:57.531674 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:57.531653    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9e5057f-96ec-4a37-a54c-6c15c320283b-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:57.533390 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:57.533358    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9e5057f-96ec-4a37-a54c-6c15c320283b-kube-api-access-v2cc5" pod "c9e5057f-96ec-4a37-a54c-6c15c320283b" (UID: "c9e5057f-96ec-4a37-a54c-6c15c320283b"). InnerVolumeSpecName "kube-api-access-v2cc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:28:57.533390 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:57.533360    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9e5057f-96ec-4a37-a54c-6c15c320283b-maas-api-xks-ca" pod "c9e5057f-96ec-4a37-a54c-6c15c320283b" (UID: "c9e5057f-96ec-4a37-a54c-6c15c320283b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:28:57.633203 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:57.633112    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9e5057f-96ec-4a37-a54c-6c15c320283b-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:57.633203 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:57.633144    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2cc5\" (UniqueName: \"kubernetes.io/projected/c9e5057f-96ec-4a37-a54c-6c15c320283b-kube-api-access-v2cc5\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:28:58.272563 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:58.272532    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2" event={"ID":"c9e5057f-96ec-4a37-a54c-6c15c320283b","Type":"ContainerDied","Data":"6db2291f0c1e7aba9d0c0c716e641e41bde185282fc33d927af8bbb3eec8f0c8"}
Oct 30 23:28:58.272563 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:58.272561    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2"
Oct 30 23:28:58.272755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:58.272577    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6db2291f0c1e7aba9d0c0c716e641e41bde185282fc33d927af8bbb3eec8f0c8"
Oct 30 23:28:58.298974 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:58.298937    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2278116c-cd84-4312-999a-832dae090303" path="/var/lib/kubelet/pods/2278116c-cd84-4312-999a-832dae090303/volumes"
Oct 30 23:28:58.299460 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:58.299438    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74c1fe84-7d11-49a3-b714-198724453ebc" path="/var/lib/kubelet/pods/74c1fe84-7d11-49a3-b714-198724453ebc/volumes"
Oct 30 23:28:59.048586 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.048554    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4"]
Oct 30 23:28:59.098330 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.098298    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-109eace1-w9df4"]
Oct 30 23:28:59.508680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.508584    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"]
Oct 30 23:28:59.509184 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.509171    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c9e5057f-96ec-4a37-a54c-6c15c320283b" containerName="revoke-keys"
Oct 30 23:28:59.509229 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.509185    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9e5057f-96ec-4a37-a54c-6c15c320283b" containerName="revoke-keys"
Oct 30 23:28:59.509268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.509260    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9e5057f-96ec-4a37-a54c-6c15c320283b" containerName="revoke-keys"
Oct 30 23:28:59.513939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.513919    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.558439 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.558413    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.558577 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.558459    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.558577 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.558520    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xlxl\" (UniqueName: \"kubernetes.io/projected/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-kube-api-access-4xlxl\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.558577 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.558572    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.558763 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.558611    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.558808 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.558786    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.558845 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.558831    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.558921 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.558909    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.558964 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.558952    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.660724 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.660681    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.660909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.660742    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.660909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.660821    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.660909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.660866    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.661084 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.660922    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.661084 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.660967    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.661084 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.660991    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xlxl\" (UniqueName: \"kubernetes.io/projected/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-kube-api-access-4xlxl\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.661084 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.661068    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.661280 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.661109    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-data\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.661280 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.661113    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.661280 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.661191    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-workload-certs\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.661393 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.661292    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-credential-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.661523 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.661500    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-workload-socket\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.661900 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.661829    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istiod-ca-cert\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.663342 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.663319    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-envoy\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.663548 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.663531    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-podinfo\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.720845 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.720784    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-token\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.722962 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.722927    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xlxl\" (UniqueName: \"kubernetes.io/projected/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-kube-api-access-4xlxl\") pod \"e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") " pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:28:59.728229 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.728197    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"]
Oct 30 23:28:59.828277 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:28:59.828251    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:29:00.119093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.119016    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"]
Oct 30 23:29:00.124379 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.124358    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.167751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.167725    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"]
Oct 30 23:29:00.169363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.169340    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-data\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.169482 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.169374    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-workload-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.169482 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.169407    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-podinfo\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.169584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.169553    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-token\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.169647 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.169583    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr4km\" (UniqueName: \"kubernetes.io/projected/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-kube-api-access-tr4km\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.169814 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.169794    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-credential-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.169858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.169844    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-workload-certs\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.169968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.169951    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istiod-ca-cert\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.170023 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.170011    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-envoy\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.171909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.171894    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.188640 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.188617    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"]
Oct 30 23:29:00.238526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.238492    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"]
Oct 30 23:29:00.271828 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.271792    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-podinfo\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.272013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.271832    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-data\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.272013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.271875    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-credential-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.272013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.271998    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-token\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.272179 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.272129    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-workload-certs\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.272224 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.272197    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istiod-ca-cert\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.272278 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.272243    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-data\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.272278 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.272262    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-data\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.272391 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.272304    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-workload-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.272391 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.272379    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-envoy\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.272492 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.272426    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88bd702-4a73-4443-9dce-eed8bde3c933-istiod-ca-cert\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.272492 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.272485    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvwls\" (UniqueName: \"kubernetes.io/projected/e88bd702-4a73-4443-9dce-eed8bde3c933-kube-api-access-vvwls\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.272631 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.272528    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-credential-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.272631 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.272554    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-token\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.272631 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.272556    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-workload-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.272631 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.272613    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr4km\" (UniqueName: \"kubernetes.io/projected/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-kube-api-access-tr4km\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.272852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.272658    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-workload-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.272852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.272730    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-podinfo\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.272852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.272765    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-workload-certs\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.272852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.272810    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-envoy\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.273063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.273045    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istiod-ca-cert\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.273345 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.273206    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-credential-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.273345 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.273226    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-workload-certs\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.275415 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.275390    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-podinfo\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.275737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.275719    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-envoy\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.289320 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:29:00.289291    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd03d3b6_2d7b_4f65_97c8_a4ea4384d80a.slice/crio-10650232309a3b097b202e43d8c79466a2565d160a91ed0b17bbaf1728af3f11 WatchSource:0}: Error finding container 10650232309a3b097b202e43d8c79466a2565d160a91ed0b17bbaf1728af3f11: Status 404 returned error can't find the container with id 10650232309a3b097b202e43d8c79466a2565d160a91ed0b17bbaf1728af3f11
Oct 30 23:29:00.292189 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.292158    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:29:00.292260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.292220    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:29:00.292260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.292248    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:29:00.300763 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.300741    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d23c612f-7a03-4b86-af74-6315ddf4657a" path="/var/lib/kubelet/pods/d23c612f-7a03-4b86-af74-6315ddf4657a/volumes"
Oct 30 23:29:00.301073 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.301060    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"]
Oct 30 23:29:00.329949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.329927    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-token\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.330037 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.329923    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr4km\" (UniqueName: \"kubernetes.io/projected/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-kube-api-access-tr4km\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.374567 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.374502    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-data\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.374728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.374658    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-envoy\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.374728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.374710    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88bd702-4a73-4443-9dce-eed8bde3c933-istiod-ca-cert\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.374853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.374811    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvwls\" (UniqueName: \"kubernetes.io/projected/e88bd702-4a73-4443-9dce-eed8bde3c933-kube-api-access-vvwls\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.374909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.374863    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-data\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.374909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.374871    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-workload-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.375020 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.374988    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-podinfo\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.375073 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.375018    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-workload-certs\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.376065 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.375313    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-credential-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.376065 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.375368    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-token\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.376065 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.375450    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-workload-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.376065 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.375763    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-credential-socket\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.376065 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.375777    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-workload-certs\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.376484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.376461    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88bd702-4a73-4443-9dce-eed8bde3c933-istiod-ca-cert\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.377155 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.377136    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-envoy\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.378118 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.378092    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-podinfo\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.432204 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.432166    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-token\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.432914 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.432888    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvwls\" (UniqueName: \"kubernetes.io/projected/e88bd702-4a73-4443-9dce-eed8bde3c933-kube-api-access-vvwls\") pod \"e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") " pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.439106 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.439087    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:00.484940 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.484903    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:00.932528 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:29:00.932495    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26dfc3f5_4b7f_4524_bd85_18df3da28f8d.slice/crio-7548176ec59ade9a5c235c9d6b75dd81ae90a801b50e5df85b1feb6c94949b99 WatchSource:0}: Error finding container 7548176ec59ade9a5c235c9d6b75dd81ae90a801b50e5df85b1feb6c94949b99: Status 404 returned error can't find the container with id 7548176ec59ade9a5c235c9d6b75dd81ae90a801b50e5df85b1feb6c94949b99
Oct 30 23:29:00.932781 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.932762    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"]
Oct 30 23:29:00.935484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.935450    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:29:00.935633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.935530    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:29:00.935633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:00.935576    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:29:01.134426 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:29:01.134373    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88bd702_4a73_4443_9dce_eed8bde3c933.slice/crio-aef5c33a26f2ce3c1c18347285d233b4838866c8068c5ecee1e2edd32db7c78a WatchSource:0}: Error finding container aef5c33a26f2ce3c1c18347285d233b4838866c8068c5ecee1e2edd32db7c78a: Status 404 returned error can't find the container with id aef5c33a26f2ce3c1c18347285d233b4838866c8068c5ecee1e2edd32db7c78a
Oct 30 23:29:01.137073 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.137042    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:29:01.137174 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.137109    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:29:01.137174 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.137124    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"]
Oct 30 23:29:01.137174 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.137149    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:29:01.304064 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.304034    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n" event={"ID":"26dfc3f5-4b7f-4524-bd85-18df3da28f8d","Type":"ContainerStarted","Data":"cfb9c21ce59bb3bfe5495e4d4199165f1fd58ae11f96414a97f5dd3955ce3a3b"}
Oct 30 23:29:01.304273 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.304084    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n" event={"ID":"26dfc3f5-4b7f-4524-bd85-18df3da28f8d","Type":"ContainerStarted","Data":"7548176ec59ade9a5c235c9d6b75dd81ae90a801b50e5df85b1feb6c94949b99"}
Oct 30 23:29:01.308537 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.308504    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4" event={"ID":"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a","Type":"ContainerStarted","Data":"41bab5387a9738ebcff4064abce5a1476516da88b53d79a2c6ec4cda43295493"}
Oct 30 23:29:01.308676 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.308567    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4" event={"ID":"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a","Type":"ContainerStarted","Data":"10650232309a3b097b202e43d8c79466a2565d160a91ed0b17bbaf1728af3f11"}
Oct 30 23:29:01.311807 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.311779    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p" event={"ID":"e88bd702-4a73-4443-9dce-eed8bde3c933","Type":"ContainerStarted","Data":"115499743e8d570c40a55ce0df5bfda246b58bf7fe02690f7b42e1953e42f9e0"}
Oct 30 23:29:01.311927 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.311831    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p" event={"ID":"e88bd702-4a73-4443-9dce-eed8bde3c933","Type":"ContainerStarted","Data":"aef5c33a26f2ce3c1c18347285d233b4838866c8068c5ecee1e2edd32db7c78a"}
Oct 30 23:29:01.432567 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.432178    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n" podStartSLOduration=1.432159911 podStartE2EDuration="1.432159911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:29:01.427068183 +0000 UTC m=+1327.314974528" watchObservedRunningTime="2026-10-30 23:29:01.432159911 +0000 UTC m=+1327.320066265"
Oct 30 23:29:01.440017 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.439989    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:01.485700 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.485617    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:01.553484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.553425    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4" podStartSLOduration=2.55340536 podStartE2EDuration="2.55340536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:28:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:29:01.548292831 +0000 UTC m=+1327.436199186" watchObservedRunningTime="2026-10-30 23:29:01.55340536 +0000 UTC m=+1327.441311717"
Oct 30 23:29:01.688623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.687630    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p" podStartSLOduration=1.687611427 podStartE2EDuration="1.687611427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:29:01.686233311 +0000 UTC m=+1327.574139677" watchObservedRunningTime="2026-10-30 23:29:01.687611427 +0000 UTC m=+1327.575517772"
Oct 30 23:29:01.829082 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.829050    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:29:01.830048 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.830023    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 23:29:01.830181 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.830073    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4" podUID="dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 23:29:01.912946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.912905    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm"]
Oct 30 23:29:01.934620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.932990    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm"
Oct 30 23:29:01.954056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:01.953984    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm"]
Oct 30 23:29:02.039622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.013989    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:29:02.039622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.014793    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:29:02.039622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.014876    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:29:02.039622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.014944    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:29:02.041649 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.040020    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f88243-cb68-41f2-b187-79830eb3d662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f772e68d-c2wbm\" (UID: \"32f88243-cb68-41f2-b187-79830eb3d662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm"
Oct 30 23:29:02.041649 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.040171    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f88243-cb68-41f2-b187-79830eb3d662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f772e68d-c2wbm\" (UID: \"32f88243-cb68-41f2-b187-79830eb3d662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm"
Oct 30 23:29:02.041649 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.040212    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwwpd\" (UniqueName: \"kubernetes.io/projected/32f88243-cb68-41f2-b187-79830eb3d662-kube-api-access-wwwpd\") pod \"maas-api-revoke-sub-f772e68d-c2wbm\" (UID: \"32f88243-cb68-41f2-b187-79830eb3d662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm"
Oct 30 23:29:02.141474 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.141379    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f88243-cb68-41f2-b187-79830eb3d662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f772e68d-c2wbm\" (UID: \"32f88243-cb68-41f2-b187-79830eb3d662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm"
Oct 30 23:29:02.141954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.141504    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f88243-cb68-41f2-b187-79830eb3d662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f772e68d-c2wbm\" (UID: \"32f88243-cb68-41f2-b187-79830eb3d662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm"
Oct 30 23:29:02.141954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.141531    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwwpd\" (UniqueName: \"kubernetes.io/projected/32f88243-cb68-41f2-b187-79830eb3d662-kube-api-access-wwwpd\") pod \"maas-api-revoke-sub-f772e68d-c2wbm\" (UID: \"32f88243-cb68-41f2-b187-79830eb3d662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm"
Oct 30 23:29:02.142371 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.142349    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f88243-cb68-41f2-b187-79830eb3d662-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f772e68d-c2wbm\" (UID: \"32f88243-cb68-41f2-b187-79830eb3d662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm"
Oct 30 23:29:02.145432 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.145393    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f88243-cb68-41f2-b187-79830eb3d662-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f772e68d-c2wbm\" (UID: \"32f88243-cb68-41f2-b187-79830eb3d662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm"
Oct 30 23:29:02.222281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.222209    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwwpd\" (UniqueName: \"kubernetes.io/projected/32f88243-cb68-41f2-b187-79830eb3d662-kube-api-access-wwwpd\") pod \"maas-api-revoke-sub-f772e68d-c2wbm\" (UID: \"32f88243-cb68-41f2-b187-79830eb3d662\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm"
Oct 30 23:29:02.301622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.285852    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm"
Oct 30 23:29:02.442690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.442467    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 23:29:02.442690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.442527    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n" podUID="26dfc3f5-4b7f-4524-bd85-18df3da28f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 23:29:02.501623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.489092    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 23:29:02.501623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.489153    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p" podUID="e88bd702-4a73-4443-9dce-eed8bde3c933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 23:29:02.841059 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.841028    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 23:29:02.841318 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.841286    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4" podUID="dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 23:29:02.988959 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:02.962647    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm"]
Oct 30 23:29:03.452869 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:03.452830    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 23:29:03.453346 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:03.452902    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n" podUID="26dfc3f5-4b7f-4524-bd85-18df3da28f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 23:29:03.500712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:03.496971    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 23:29:03.500712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:03.497092    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p" podUID="e88bd702-4a73-4443-9dce-eed8bde3c933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 23:29:03.612831 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:03.612737    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm" event={"ID":"32f88243-cb68-41f2-b187-79830eb3d662","Type":"ContainerStarted","Data":"e5789b572d1eaa74e2bb50274032b80a764b261efb4614b4ddbc10b6b9affbe1"}
Oct 30 23:29:03.833375 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:03.829294    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 23:29:03.833375 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:03.829354    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4" podUID="dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 23:29:04.039326 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.039289    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h"]
Oct 30 23:29:04.043542 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.043526    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h"
Oct 30 23:29:04.151698 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.148798    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h"]
Oct 30 23:29:04.234623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.199183    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4ffr\" (UniqueName: \"kubernetes.io/projected/76ec7768-8590-4a51-8c3a-e47d659b0d56-kube-api-access-m4ffr\") pod \"maas-api-revoke-sub-61525f73-jlx5h\" (UID: \"76ec7768-8590-4a51-8c3a-e47d659b0d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h"
Oct 30 23:29:04.234623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.199278    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ec7768-8590-4a51-8c3a-e47d659b0d56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61525f73-jlx5h\" (UID: \"76ec7768-8590-4a51-8c3a-e47d659b0d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h"
Oct 30 23:29:04.234623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.231762    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ec7768-8590-4a51-8c3a-e47d659b0d56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61525f73-jlx5h\" (UID: \"76ec7768-8590-4a51-8c3a-e47d659b0d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h"
Oct 30 23:29:04.341892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.341764    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4ffr\" (UniqueName: \"kubernetes.io/projected/76ec7768-8590-4a51-8c3a-e47d659b0d56-kube-api-access-m4ffr\") pod \"maas-api-revoke-sub-61525f73-jlx5h\" (UID: \"76ec7768-8590-4a51-8c3a-e47d659b0d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h"
Oct 30 23:29:04.341892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.341848    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ec7768-8590-4a51-8c3a-e47d659b0d56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61525f73-jlx5h\" (UID: \"76ec7768-8590-4a51-8c3a-e47d659b0d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h"
Oct 30 23:29:04.342165 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.341925    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ec7768-8590-4a51-8c3a-e47d659b0d56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61525f73-jlx5h\" (UID: \"76ec7768-8590-4a51-8c3a-e47d659b0d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h"
Oct 30 23:29:04.344638 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.342775    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ec7768-8590-4a51-8c3a-e47d659b0d56-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61525f73-jlx5h\" (UID: \"76ec7768-8590-4a51-8c3a-e47d659b0d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h"
Oct 30 23:29:04.347430 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.347055    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ec7768-8590-4a51-8c3a-e47d659b0d56-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61525f73-jlx5h\" (UID: \"76ec7768-8590-4a51-8c3a-e47d659b0d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h"
Oct 30 23:29:04.440040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.439918    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 23:29:04.440386 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.440283    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n" podUID="26dfc3f5-4b7f-4524-bd85-18df3da28f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 23:29:04.442469 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.442436    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4ffr\" (UniqueName: \"kubernetes.io/projected/76ec7768-8590-4a51-8c3a-e47d659b0d56-kube-api-access-m4ffr\") pod \"maas-api-revoke-sub-61525f73-jlx5h\" (UID: \"76ec7768-8590-4a51-8c3a-e47d659b0d56\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h"
Oct 30 23:29:04.492622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.490188    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 23:29:04.492622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.490248    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p" podUID="e88bd702-4a73-4443-9dce-eed8bde3c933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 23:29:04.653867 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.650849    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm" event={"ID":"32f88243-cb68-41f2-b187-79830eb3d662","Type":"ContainerStarted","Data":"ef81c2a1084c193a898e06dfd054906b097221921ff53710b58881d9527f5579"}
Oct 30 23:29:04.695671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.676820    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h"
Oct 30 23:29:04.841941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.841867    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 23:29:04.841941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:04.841931    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4" podUID="dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 23:29:05.137487 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.137453    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs"]
Oct 30 23:29:05.147964 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.147939    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs"
Oct 30 23:29:05.151734 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.151708    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs"]
Oct 30 23:29:05.162798 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.160881    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-564d5020-lswjs\" (UID: \"64d12e01-68f8-431f-8ad7-69d6eb74bbbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs"
Oct 30 23:29:05.162798 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.160986    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68bnm\" (UniqueName: \"kubernetes.io/projected/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-kube-api-access-68bnm\") pod \"maas-api-revoke-sub-564d5020-lswjs\" (UID: \"64d12e01-68f8-431f-8ad7-69d6eb74bbbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs"
Oct 30 23:29:05.162798 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.161039    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-564d5020-lswjs\" (UID: \"64d12e01-68f8-431f-8ad7-69d6eb74bbbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs"
Oct 30 23:29:05.329625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.311704    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-564d5020-lswjs\" (UID: \"64d12e01-68f8-431f-8ad7-69d6eb74bbbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs"
Oct 30 23:29:05.329625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.311834    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68bnm\" (UniqueName: \"kubernetes.io/projected/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-kube-api-access-68bnm\") pod \"maas-api-revoke-sub-564d5020-lswjs\" (UID: \"64d12e01-68f8-431f-8ad7-69d6eb74bbbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs"
Oct 30 23:29:05.329625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.311888    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-564d5020-lswjs\" (UID: \"64d12e01-68f8-431f-8ad7-69d6eb74bbbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs"
Oct 30 23:29:05.329625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.312774    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-564d5020-lswjs\" (UID: \"64d12e01-68f8-431f-8ad7-69d6eb74bbbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs"
Oct 30 23:29:05.353648 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.350939    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h"]
Oct 30 23:29:05.445638 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.444498    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 23:29:05.445638 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.444540    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n" podUID="26dfc3f5-4b7f-4524-bd85-18df3da28f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 23:29:05.445638 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.444658    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68bnm\" (UniqueName: \"kubernetes.io/projected/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-kube-api-access-68bnm\") pod \"maas-api-revoke-sub-564d5020-lswjs\" (UID: \"64d12e01-68f8-431f-8ad7-69d6eb74bbbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs"
Oct 30 23:29:05.445638 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.444984    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-564d5020-lswjs\" (UID: \"64d12e01-68f8-431f-8ad7-69d6eb74bbbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs"
Oct 30 23:29:05.497623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.496519    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 23:29:05.497623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.496577    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p" podUID="e88bd702-4a73-4443-9dce-eed8bde3c933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 23:29:05.549512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.549480    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs"
Oct 30 23:29:05.755470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.755432    5078 generic.go:356] "Generic (PLEG): container finished" podID="32f88243-cb68-41f2-b187-79830eb3d662" containerID="ef81c2a1084c193a898e06dfd054906b097221921ff53710b58881d9527f5579" exitCode=0
Oct 30 23:29:05.755835 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.755788    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm" event={"ID":"32f88243-cb68-41f2-b187-79830eb3d662","Type":"ContainerDied","Data":"ef81c2a1084c193a898e06dfd054906b097221921ff53710b58881d9527f5579"}
Oct 30 23:29:05.853134 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.853088    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h" event={"ID":"76ec7768-8590-4a51-8c3a-e47d659b0d56","Type":"ContainerStarted","Data":"107686da76143be1509aa7c1d490cb47eed98f1d53a50bdde691731d4b5ed45d"}
Oct 30 23:29:05.855067 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.855038    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 23:29:05.855298 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:05.855268    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4" podUID="dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 23:29:06.084659 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:06.078648    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk"]
Oct 30 23:29:06.155750 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:06.147531    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5ab077b-76nzk"]
Oct 30 23:29:06.351427 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:06.351347    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="999c0695-1f53-4b34-8331-4f569a986957" path="/var/lib/kubelet/pods/999c0695-1f53-4b34-8331-4f569a986957/volumes"
Oct 30 23:29:06.439430 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:06.439388    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 23:29:06.439624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:06.439444    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n" podUID="26dfc3f5-4b7f-4524-bd85-18df3da28f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 23:29:06.547701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:06.523307    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 23:29:06.547701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:06.523362    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p" podUID="e88bd702-4a73-4443-9dce-eed8bde3c933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 23:29:06.745708 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:06.745681    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs"]
Oct 30 23:29:06.829323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:06.829291    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 23:29:06.829411 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:06.829354    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4" podUID="dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 23:29:06.919621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:06.914303    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs" event={"ID":"64d12e01-68f8-431f-8ad7-69d6eb74bbbc","Type":"ContainerStarted","Data":"2a3f944dfb14346114577b76ef7d4881f8717422dae33a403017f555f3043bce"}
Oct 30 23:29:06.923060 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:06.922938    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm" event={"ID":"32f88243-cb68-41f2-b187-79830eb3d662","Type":"ContainerDied","Data":"e5789b572d1eaa74e2bb50274032b80a764b261efb4614b4ddbc10b6b9affbe1"}
Oct 30 23:29:06.923060 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:06.923006    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5789b572d1eaa74e2bb50274032b80a764b261efb4614b4ddbc10b6b9affbe1"
Oct 30 23:29:06.925986 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:06.925957    5078 generic.go:356] "Generic (PLEG): container finished" podID="76ec7768-8590-4a51-8c3a-e47d659b0d56" containerID="a59207809e954a4a06d2a1832d86b66864d7b1929d464a1a8c88bd8d6590d468" exitCode=0
Oct 30 23:29:06.926089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:06.926072    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h" event={"ID":"76ec7768-8590-4a51-8c3a-e47d659b0d56","Type":"ContainerDied","Data":"a59207809e954a4a06d2a1832d86b66864d7b1929d464a1a8c88bd8d6590d468"}
Oct 30 23:29:06.928042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:06.928018    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm"
Oct 30 23:29:07.011683 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.011144    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32f88243-cb68-41f2-b187-79830eb3d662-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f88243-cb68-41f2-b187-79830eb3d662-maas-api-xks-ca\") pod \"32f88243-cb68-41f2-b187-79830eb3d662\" (UID: \"32f88243-cb68-41f2-b187-79830eb3d662\") "
Oct 30 23:29:07.011683 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.011315    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f88243-cb68-41f2-b187-79830eb3d662-kube-api-access-wwwpd\" (UniqueName: \"kubernetes.io/projected/32f88243-cb68-41f2-b187-79830eb3d662-kube-api-access-wwwpd\") pod \"32f88243-cb68-41f2-b187-79830eb3d662\" (UID: \"32f88243-cb68-41f2-b187-79830eb3d662\") "
Oct 30 23:29:07.011683 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.011476    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32f88243-cb68-41f2-b187-79830eb3d662-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f88243-cb68-41f2-b187-79830eb3d662-maas-api-openshift-ca\") pod \"32f88243-cb68-41f2-b187-79830eb3d662\" (UID: \"32f88243-cb68-41f2-b187-79830eb3d662\") "
Oct 30 23:29:07.012620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.012527    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f88243-cb68-41f2-b187-79830eb3d662-maas-api-openshift-ca" pod "32f88243-cb68-41f2-b187-79830eb3d662" (UID: "32f88243-cb68-41f2-b187-79830eb3d662"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:29:07.019002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.018965    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f88243-cb68-41f2-b187-79830eb3d662-kube-api-access-wwwpd" pod "32f88243-cb68-41f2-b187-79830eb3d662" (UID: "32f88243-cb68-41f2-b187-79830eb3d662"). InnerVolumeSpecName "kube-api-access-wwwpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:29:07.019002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.018992    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32f88243-cb68-41f2-b187-79830eb3d662-maas-api-xks-ca" pod "32f88243-cb68-41f2-b187-79830eb3d662" (UID: "32f88243-cb68-41f2-b187-79830eb3d662"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:29:07.116947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.116499    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwwpd\" (UniqueName: \"kubernetes.io/projected/32f88243-cb68-41f2-b187-79830eb3d662-kube-api-access-wwwpd\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:07.116947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.116540    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32f88243-cb68-41f2-b187-79830eb3d662-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:07.116947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.116558    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32f88243-cb68-41f2-b187-79830eb3d662-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:07.440223 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.440188    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused" start-of-body=
Oct 30 23:29:07.440379 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.440257    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n" podUID="26dfc3f5-4b7f-4524-bd85-18df3da28f8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.106:15021/healthz/ready\": dial tcp 10.133.0.106:15021: connect: connection refused"
Oct 30 23:29:07.485506 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.485478    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 23:29:07.485774 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.485750    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p" podUID="e88bd702-4a73-4443-9dce-eed8bde3c933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 23:29:07.637109 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.637019    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9"]
Oct 30 23:29:07.637655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.637633    5078 cpu_manager.go:414] "containerMap: removing container" podUID="32f88243-cb68-41f2-b187-79830eb3d662" containerName="revoke-keys"
Oct 30 23:29:07.637655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.637654    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f88243-cb68-41f2-b187-79830eb3d662" containerName="revoke-keys"
Oct 30 23:29:07.637811 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.637777    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f88243-cb68-41f2-b187-79830eb3d662" containerName="revoke-keys"
Oct 30 23:29:07.641487 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.641467    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9"
Oct 30 23:29:07.700565 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.700536    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9"]
Oct 30 23:29:07.728078 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.728033    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgc8f\" (UniqueName: \"kubernetes.io/projected/e8e48362-06af-40ad-8e4d-089ebcb9bb78-kube-api-access-kgc8f\") pod \"maas-api-revoke-sub-525485d7-p6rv9\" (UID: \"e8e48362-06af-40ad-8e4d-089ebcb9bb78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9"
Oct 30 23:29:07.728244 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.728143    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e48362-06af-40ad-8e4d-089ebcb9bb78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-525485d7-p6rv9\" (UID: \"e8e48362-06af-40ad-8e4d-089ebcb9bb78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9"
Oct 30 23:29:07.728244 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.728205    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e48362-06af-40ad-8e4d-089ebcb9bb78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-525485d7-p6rv9\" (UID: \"e8e48362-06af-40ad-8e4d-089ebcb9bb78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9"
Oct 30 23:29:07.828746 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.828708    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Oct 30 23:29:07.828939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.828771    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4" podUID="dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Oct 30 23:29:07.829477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.829444    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgc8f\" (UniqueName: \"kubernetes.io/projected/e8e48362-06af-40ad-8e4d-089ebcb9bb78-kube-api-access-kgc8f\") pod \"maas-api-revoke-sub-525485d7-p6rv9\" (UID: \"e8e48362-06af-40ad-8e4d-089ebcb9bb78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9"
Oct 30 23:29:07.829589 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.829558    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e48362-06af-40ad-8e4d-089ebcb9bb78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-525485d7-p6rv9\" (UID: \"e8e48362-06af-40ad-8e4d-089ebcb9bb78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9"
Oct 30 23:29:07.829691 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.829677    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e48362-06af-40ad-8e4d-089ebcb9bb78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-525485d7-p6rv9\" (UID: \"e8e48362-06af-40ad-8e4d-089ebcb9bb78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9"
Oct 30 23:29:07.830320 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.830297    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e48362-06af-40ad-8e4d-089ebcb9bb78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-525485d7-p6rv9\" (UID: \"e8e48362-06af-40ad-8e4d-089ebcb9bb78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9"
Oct 30 23:29:07.832887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.832857    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e48362-06af-40ad-8e4d-089ebcb9bb78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-525485d7-p6rv9\" (UID: \"e8e48362-06af-40ad-8e4d-089ebcb9bb78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9"
Oct 30 23:29:07.887195 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.887112    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgc8f\" (UniqueName: \"kubernetes.io/projected/e8e48362-06af-40ad-8e4d-089ebcb9bb78-kube-api-access-kgc8f\") pod \"maas-api-revoke-sub-525485d7-p6rv9\" (UID: \"e8e48362-06af-40ad-8e4d-089ebcb9bb78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9"
Oct 30 23:29:07.940479 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.940441    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs" event={"ID":"64d12e01-68f8-431f-8ad7-69d6eb74bbbc","Type":"ContainerStarted","Data":"2ea4be0290714ae52635e0972fae244fb82054ce74d1265345dcbcdf89784a91"}
Oct 30 23:29:07.941473 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.941120    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm"
Oct 30 23:29:07.961556 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:07.961324    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9"
Oct 30 23:29:08.106219 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.106131    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9"]
Oct 30 23:29:08.113692 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.113632    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-260604bf-sxpt9"]
Oct 30 23:29:08.115739 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.115691    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs" podStartSLOduration=3.115674793 podStartE2EDuration="3.115674793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:29:08.065041271 +0000 UTC m=+1333.952947625" watchObservedRunningTime="2026-10-30 23:29:08.115674793 +0000 UTC m=+1334.003581146"
Oct 30 23:29:08.230784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.230433    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h"
Oct 30 23:29:08.311518 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.311483    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1905425-0b5c-4ed9-85f0-b674aaafe146" path="/var/lib/kubelet/pods/c1905425-0b5c-4ed9-85f0-b674aaafe146/volumes"
Oct 30 23:29:08.344125 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.344087    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76ec7768-8590-4a51-8c3a-e47d659b0d56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ec7768-8590-4a51-8c3a-e47d659b0d56-maas-api-xks-ca\") pod \"76ec7768-8590-4a51-8c3a-e47d659b0d56\" (UID: \"76ec7768-8590-4a51-8c3a-e47d659b0d56\") "
Oct 30 23:29:08.344512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.344485    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76ec7768-8590-4a51-8c3a-e47d659b0d56-kube-api-access-m4ffr\" (UniqueName: \"kubernetes.io/projected/76ec7768-8590-4a51-8c3a-e47d659b0d56-kube-api-access-m4ffr\") pod \"76ec7768-8590-4a51-8c3a-e47d659b0d56\" (UID: \"76ec7768-8590-4a51-8c3a-e47d659b0d56\") "
Oct 30 23:29:08.344989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.344966    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76ec7768-8590-4a51-8c3a-e47d659b0d56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ec7768-8590-4a51-8c3a-e47d659b0d56-maas-api-openshift-ca\") pod \"76ec7768-8590-4a51-8c3a-e47d659b0d56\" (UID: \"76ec7768-8590-4a51-8c3a-e47d659b0d56\") "
Oct 30 23:29:08.352940 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.352911    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76ec7768-8590-4a51-8c3a-e47d659b0d56-maas-api-openshift-ca" pod "76ec7768-8590-4a51-8c3a-e47d659b0d56" (UID: "76ec7768-8590-4a51-8c3a-e47d659b0d56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:29:08.382941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.382878    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76ec7768-8590-4a51-8c3a-e47d659b0d56-kube-api-access-m4ffr" pod "76ec7768-8590-4a51-8c3a-e47d659b0d56" (UID: "76ec7768-8590-4a51-8c3a-e47d659b0d56"). InnerVolumeSpecName "kube-api-access-m4ffr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:29:08.432721 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.432682    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76ec7768-8590-4a51-8c3a-e47d659b0d56-maas-api-xks-ca" pod "76ec7768-8590-4a51-8c3a-e47d659b0d56" (UID: "76ec7768-8590-4a51-8c3a-e47d659b0d56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:29:08.475152 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.459984    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9"]
Oct 30 23:29:08.477054 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.477021    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/76ec7768-8590-4a51-8c3a-e47d659b0d56-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:08.477246 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.477228    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4ffr\" (UniqueName: \"kubernetes.io/projected/76ec7768-8590-4a51-8c3a-e47d659b0d56-kube-api-access-m4ffr\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:08.477420 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.477404    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/76ec7768-8590-4a51-8c3a-e47d659b0d56-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:08.498648 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.498443    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Oct 30 23:29:08.498648 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.498542    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p" podUID="e88bd702-4a73-4443-9dce-eed8bde3c933" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Oct 30 23:29:08.835878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.835844    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:29:08.837008 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.836962    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:29:08.838428 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.838407    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:29:08.956293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.956260    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h" event={"ID":"76ec7768-8590-4a51-8c3a-e47d659b0d56","Type":"ContainerDied","Data":"107686da76143be1509aa7c1d490cb47eed98f1d53a50bdde691731d4b5ed45d"}
Oct 30 23:29:08.956465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.956309    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="107686da76143be1509aa7c1d490cb47eed98f1d53a50bdde691731d4b5ed45d"
Oct 30 23:29:08.956465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.956373    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h"
Oct 30 23:29:08.962822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.959523    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9" event={"ID":"e8e48362-06af-40ad-8e4d-089ebcb9bb78","Type":"ContainerStarted","Data":"0674c02fba5a87566bcc6aff9cef3d2b02117feb92c5893b4e52495cea4a537d"}
Oct 30 23:29:08.962822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.959584    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9" event={"ID":"e8e48362-06af-40ad-8e4d-089ebcb9bb78","Type":"ContainerStarted","Data":"9ee92ed6129ea9778e3a36f7640ff7c713e406a3ef14341a4e549aff298cb4ff"}
Oct 30 23:29:08.962822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.962751    5078 generic.go:356] "Generic (PLEG): container finished" podID="64d12e01-68f8-431f-8ad7-69d6eb74bbbc" containerID="2ea4be0290714ae52635e0972fae244fb82054ce74d1265345dcbcdf89784a91" exitCode=0
Oct 30 23:29:08.969287 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:08.963701    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs" event={"ID":"64d12e01-68f8-431f-8ad7-69d6eb74bbbc","Type":"ContainerDied","Data":"2ea4be0290714ae52635e0972fae244fb82054ce74d1265345dcbcdf89784a91"}
Oct 30 23:29:09.000584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:09.000555    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:09.001862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:09.001840    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:09.002961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:09.002939    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:29:09.320887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:09.320835    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9" podStartSLOduration=2.320821457 podStartE2EDuration="2.320821457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:29:09.315791341 +0000 UTC m=+1335.203697706" watchObservedRunningTime="2026-10-30 23:29:09.320821457 +0000 UTC m=+1335.208727810"
Oct 30 23:29:09.490822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:09.490792    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:09.491310 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:09.491292    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:09.492202 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:09.492183    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:29:09.972000 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:09.971969    5078 generic.go:356] "Generic (PLEG): container finished" podID="e8e48362-06af-40ad-8e4d-089ebcb9bb78" containerID="0674c02fba5a87566bcc6aff9cef3d2b02117feb92c5893b4e52495cea4a537d" exitCode=0
Oct 30 23:29:09.972406 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:09.972053    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9" event={"ID":"e8e48362-06af-40ad-8e4d-089ebcb9bb78","Type":"ContainerDied","Data":"0674c02fba5a87566bcc6aff9cef3d2b02117feb92c5893b4e52495cea4a537d"}
Oct 30 23:29:10.135551 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:10.135522    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs"
Oct 30 23:29:10.211455 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:10.211425    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-maas-api-xks-ca\") pod \"64d12e01-68f8-431f-8ad7-69d6eb74bbbc\" (UID: \"64d12e01-68f8-431f-8ad7-69d6eb74bbbc\") "
Oct 30 23:29:10.211592 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:10.211529    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-kube-api-access-68bnm\" (UniqueName: \"kubernetes.io/projected/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-kube-api-access-68bnm\") pod \"64d12e01-68f8-431f-8ad7-69d6eb74bbbc\" (UID: \"64d12e01-68f8-431f-8ad7-69d6eb74bbbc\") "
Oct 30 23:29:10.211690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:10.211673    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-maas-api-openshift-ca\") pod \"64d12e01-68f8-431f-8ad7-69d6eb74bbbc\" (UID: \"64d12e01-68f8-431f-8ad7-69d6eb74bbbc\") "
Oct 30 23:29:10.212019 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:10.211992    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-maas-api-openshift-ca" pod "64d12e01-68f8-431f-8ad7-69d6eb74bbbc" (UID: "64d12e01-68f8-431f-8ad7-69d6eb74bbbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:29:10.212283 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:10.212251    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:10.213661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:10.213635    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-maas-api-xks-ca" pod "64d12e01-68f8-431f-8ad7-69d6eb74bbbc" (UID: "64d12e01-68f8-431f-8ad7-69d6eb74bbbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:29:10.213755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:10.213731    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-kube-api-access-68bnm" pod "64d12e01-68f8-431f-8ad7-69d6eb74bbbc" (UID: "64d12e01-68f8-431f-8ad7-69d6eb74bbbc"). InnerVolumeSpecName "kube-api-access-68bnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:29:10.313746 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:10.313720    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68bnm\" (UniqueName: \"kubernetes.io/projected/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-kube-api-access-68bnm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:10.313746 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:10.313749    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64d12e01-68f8-431f-8ad7-69d6eb74bbbc-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:10.981507 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:10.981475    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs"
Oct 30 23:29:10.981932 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:10.981498    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs" event={"ID":"64d12e01-68f8-431f-8ad7-69d6eb74bbbc","Type":"ContainerDied","Data":"2a3f944dfb14346114577b76ef7d4881f8717422dae33a403017f555f3043bce"}
Oct 30 23:29:10.981932 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:10.981579    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a3f944dfb14346114577b76ef7d4881f8717422dae33a403017f555f3043bce"
Oct 30 23:29:11.145293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.145270    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9"
Oct 30 23:29:11.232619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.232541    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8e48362-06af-40ad-8e4d-089ebcb9bb78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e48362-06af-40ad-8e4d-089ebcb9bb78-maas-api-xks-ca\") pod \"e8e48362-06af-40ad-8e4d-089ebcb9bb78\" (UID: \"e8e48362-06af-40ad-8e4d-089ebcb9bb78\") "
Oct 30 23:29:11.232754 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.232740    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8e48362-06af-40ad-8e4d-089ebcb9bb78-kube-api-access-kgc8f\" (UniqueName: \"kubernetes.io/projected/e8e48362-06af-40ad-8e4d-089ebcb9bb78-kube-api-access-kgc8f\") pod \"e8e48362-06af-40ad-8e4d-089ebcb9bb78\" (UID: \"e8e48362-06af-40ad-8e4d-089ebcb9bb78\") "
Oct 30 23:29:11.232792 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.232780    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8e48362-06af-40ad-8e4d-089ebcb9bb78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e48362-06af-40ad-8e4d-089ebcb9bb78-maas-api-openshift-ca\") pod \"e8e48362-06af-40ad-8e4d-089ebcb9bb78\" (UID: \"e8e48362-06af-40ad-8e4d-089ebcb9bb78\") "
Oct 30 23:29:11.233195 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.233172    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8e48362-06af-40ad-8e4d-089ebcb9bb78-maas-api-openshift-ca" pod "e8e48362-06af-40ad-8e4d-089ebcb9bb78" (UID: "e8e48362-06af-40ad-8e4d-089ebcb9bb78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:29:11.234563 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.234544    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8e48362-06af-40ad-8e4d-089ebcb9bb78-maas-api-xks-ca" pod "e8e48362-06af-40ad-8e4d-089ebcb9bb78" (UID: "e8e48362-06af-40ad-8e4d-089ebcb9bb78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:29:11.234891 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.234872    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8e48362-06af-40ad-8e4d-089ebcb9bb78-kube-api-access-kgc8f" pod "e8e48362-06af-40ad-8e4d-089ebcb9bb78" (UID: "e8e48362-06af-40ad-8e4d-089ebcb9bb78"). InnerVolumeSpecName "kube-api-access-kgc8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:29:11.274919 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.274888    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b"]
Oct 30 23:29:11.275823 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.275802    5078 cpu_manager.go:414] "containerMap: removing container" podUID="64d12e01-68f8-431f-8ad7-69d6eb74bbbc" containerName="revoke-keys"
Oct 30 23:29:11.275892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.275827    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d12e01-68f8-431f-8ad7-69d6eb74bbbc" containerName="revoke-keys"
Oct 30 23:29:11.275892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.275843    5078 cpu_manager.go:414] "containerMap: removing container" podUID="76ec7768-8590-4a51-8c3a-e47d659b0d56" containerName="revoke-keys"
Oct 30 23:29:11.275892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.275851    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76ec7768-8590-4a51-8c3a-e47d659b0d56" containerName="revoke-keys"
Oct 30 23:29:11.276027 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.275893    5078 cpu_manager.go:414] "containerMap: removing container" podUID="e8e48362-06af-40ad-8e4d-089ebcb9bb78" containerName="revoke-keys"
Oct 30 23:29:11.276027 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.275902    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8e48362-06af-40ad-8e4d-089ebcb9bb78" containerName="revoke-keys"
Oct 30 23:29:11.276107 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.276093    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="76ec7768-8590-4a51-8c3a-e47d659b0d56" containerName="revoke-keys"
Oct 30 23:29:11.276139 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.276115    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8e48362-06af-40ad-8e4d-089ebcb9bb78" containerName="revoke-keys"
Oct 30 23:29:11.276139 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.276135    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d12e01-68f8-431f-8ad7-69d6eb74bbbc" containerName="revoke-keys"
Oct 30 23:29:11.280257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.280235    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b"
Oct 30 23:29:11.333303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.333270    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b"]
Oct 30 23:29:11.333997 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.333973    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-279dfbc3-t697b\" (UID: \"75e243a4-d9b3-48c3-a5e3-a548c2ea253d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b"
Oct 30 23:29:11.334184 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.334148    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-279dfbc3-t697b\" (UID: \"75e243a4-d9b3-48c3-a5e3-a548c2ea253d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b"
Oct 30 23:29:11.334494 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.334465    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkwsl\" (UniqueName: \"kubernetes.io/projected/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-kube-api-access-kkwsl\") pod \"maas-api-revoke-sub-279dfbc3-t697b\" (UID: \"75e243a4-d9b3-48c3-a5e3-a548c2ea253d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b"
Oct 30 23:29:11.334660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.334642    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8e48362-06af-40ad-8e4d-089ebcb9bb78-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:11.334715 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.334672    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8e48362-06af-40ad-8e4d-089ebcb9bb78-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:11.334715 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.334688    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgc8f\" (UniqueName: \"kubernetes.io/projected/e8e48362-06af-40ad-8e4d-089ebcb9bb78-kube-api-access-kgc8f\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:11.436349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.436322    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkwsl\" (UniqueName: \"kubernetes.io/projected/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-kube-api-access-kkwsl\") pod \"maas-api-revoke-sub-279dfbc3-t697b\" (UID: \"75e243a4-d9b3-48c3-a5e3-a548c2ea253d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b"
Oct 30 23:29:11.436513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.436414    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-279dfbc3-t697b\" (UID: \"75e243a4-d9b3-48c3-a5e3-a548c2ea253d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b"
Oct 30 23:29:11.436513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.436479    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-279dfbc3-t697b\" (UID: \"75e243a4-d9b3-48c3-a5e3-a548c2ea253d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b"
Oct 30 23:29:11.437239 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.437212    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-279dfbc3-t697b\" (UID: \"75e243a4-d9b3-48c3-a5e3-a548c2ea253d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b"
Oct 30 23:29:11.439020 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.439000    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-279dfbc3-t697b\" (UID: \"75e243a4-d9b3-48c3-a5e3-a548c2ea253d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b"
Oct 30 23:29:11.492942 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.492883    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkwsl\" (UniqueName: \"kubernetes.io/projected/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-kube-api-access-kkwsl\") pod \"maas-api-revoke-sub-279dfbc3-t697b\" (UID: \"75e243a4-d9b3-48c3-a5e3-a548c2ea253d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b"
Oct 30 23:29:11.595684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.595647    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b"
Oct 30 23:29:11.971853 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:29:11.971825    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e243a4_d9b3_48c3_a5e3_a548c2ea253d.slice/crio-1db60b716db59832b94b4bf729f2907962c61589874fe978f12488004e7084b8 WatchSource:0}: Error finding container 1db60b716db59832b94b4bf729f2907962c61589874fe978f12488004e7084b8: Status 404 returned error can't find the container with id 1db60b716db59832b94b4bf729f2907962c61589874fe978f12488004e7084b8
Oct 30 23:29:11.972721 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.972699    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b"]
Oct 30 23:29:11.991546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.991526    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9"
Oct 30 23:29:11.991871 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.991561    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9" event={"ID":"e8e48362-06af-40ad-8e4d-089ebcb9bb78","Type":"ContainerDied","Data":"9ee92ed6129ea9778e3a36f7640ff7c713e406a3ef14341a4e549aff298cb4ff"}
Oct 30 23:29:11.991871 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.991632    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ee92ed6129ea9778e3a36f7640ff7c713e406a3ef14341a4e549aff298cb4ff"
Oct 30 23:29:11.993216 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:11.993118    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b" event={"ID":"75e243a4-d9b3-48c3-a5e3-a548c2ea253d","Type":"ContainerStarted","Data":"1db60b716db59832b94b4bf729f2907962c61589874fe978f12488004e7084b8"}
Oct 30 23:29:12.049111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:12.049086    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w"]
Oct 30 23:29:12.057717 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:12.057694    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbb516cb-t584w"]
Oct 30 23:29:12.299159 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:12.299129    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e829e1-3829-40dd-93d2-ef0d55b74127" path="/var/lib/kubelet/pods/51e829e1-3829-40dd-93d2-ef0d55b74127/volumes"
Oct 30 23:29:13.121639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:13.120441    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b" event={"ID":"75e243a4-d9b3-48c3-a5e3-a548c2ea253d","Type":"ContainerStarted","Data":"0f4551819a075b72f8d0bb5ee020de67f82c33e774f939facd75229f2718cffe"}
Oct 30 23:29:13.260622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:13.259587    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b" podStartSLOduration=2.259562123 podStartE2EDuration="2.259562123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:29:13.248027682 +0000 UTC m=+1339.135934040" watchObservedRunningTime="2026-10-30 23:29:13.259562123 +0000 UTC m=+1339.147468478"
Oct 30 23:29:14.156179 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:14.156140    5078 generic.go:356] "Generic (PLEG): container finished" podID="75e243a4-d9b3-48c3-a5e3-a548c2ea253d" containerID="0f4551819a075b72f8d0bb5ee020de67f82c33e774f939facd75229f2718cffe" exitCode=0
Oct 30 23:29:14.156867 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:14.156843    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b" event={"ID":"75e243a4-d9b3-48c3-a5e3-a548c2ea253d","Type":"ContainerDied","Data":"0f4551819a075b72f8d0bb5ee020de67f82c33e774f939facd75229f2718cffe"}
Oct 30 23:29:14.334673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:14.327696    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:14.344620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:14.341678    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:14.454682 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:14.446272    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded"
Oct 30 23:29:14.482629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:14.480721    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:14.515635 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:14.486699    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:14.515635 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:14.506846    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:14.542684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:14.531858    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 23:29:14.542684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:14.535413    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:14.601635 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:14.598741    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:14.617658 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:14.612722    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:14.781681 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:14.777954    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:14.934195 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:14.934150    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:14.947628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:14.943068    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942"]
Oct 30 23:29:15.056952 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.056905    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:15.065788 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.065761    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942"
Oct 30 23:29:15.161580 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.161540    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0340ff8-29942\" (UID: \"0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942"
Oct 30 23:29:15.299760 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.299725    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0340ff8-29942\" (UID: \"0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942"
Oct 30 23:29:15.300190 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.300163    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n68fp\" (UniqueName: \"kubernetes.io/projected/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-kube-api-access-n68fp\") pod \"maas-api-revoke-sub-b0340ff8-29942\" (UID: \"0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942"
Oct 30 23:29:15.300474 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.262156    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:15.300782 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.275311    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:15.315901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.315817    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942"]
Oct 30 23:29:15.356733 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.356694    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:15.384161 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.382914    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:29:15.421210 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.420511    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0340ff8-29942\" (UID: \"0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942"
Oct 30 23:29:15.421406 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.421354    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0340ff8-29942\" (UID: \"0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942"
Oct 30 23:29:15.460623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.444839    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0340ff8-29942\" (UID: \"0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942"
Oct 30 23:29:15.460623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.445080    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n68fp\" (UniqueName: \"kubernetes.io/projected/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-kube-api-access-n68fp\") pod \"maas-api-revoke-sub-b0340ff8-29942\" (UID: \"0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942"
Oct 30 23:29:15.460623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.446984    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:15.466201 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.466162    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:29:15.488785 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.488747    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Oct 30 23:29:15.500810 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.499676    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:15.509656 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.505899    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:29:15.509656 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.505973    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:29:15.509656 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.508540    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:15.548621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.513684    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:29:15.550459 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.549144    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:15.626250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.550718    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:29:15.626250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.551702    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 23:29:15.626250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.561717    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:29:15.626250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.599735    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:15.626250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.601396    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n68fp\" (UniqueName: \"kubernetes.io/projected/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-kube-api-access-n68fp\") pod \"maas-api-revoke-sub-b0340ff8-29942\" (UID: \"0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942"
Oct 30 23:29:15.626250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.601489    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:29:15.626250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.623012    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:15.626250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.624256    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:29:15.637161 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.637097    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk"]
Oct 30 23:29:15.793055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.792865    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 23:29:15.801744 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.801716    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:29:15.844739 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.844689    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:15.862797 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.844797    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded"
Oct 30 23:29:15.972689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.960942    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:15.972689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:15.961717    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:29:16.014834 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:16.014760    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bff4dcb-sdlqk"]
Oct 30 23:29:16.059622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:16.058746    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 23:29:16.064629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:16.060685    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:29:16.388635 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:16.376690    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:16.389366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:16.389324    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 23:29:16.465637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:16.392296    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0340ff8-29942\" (UID: \"0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942"
Oct 30 23:29:16.465637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:16.451402    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded"
Oct 30 23:29:16.528624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:16.525050    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:16.787224 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:16.787063    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:16.865633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:16.821436    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 23:29:16.865633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:16.821880    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 23:29:16.865633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:16.826806    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:16.865633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:16.805724    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 30 23:29:16.978659 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:16.528895    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:17.040766 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:17.029919    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942"
Oct 30 23:29:17.103671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:17.103635    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="411a9ab4-493f-4e0f-9e11-b2fb0db1618d" path="/var/lib/kubelet/pods/411a9ab4-493f-4e0f-9e11-b2fb0db1618d/volumes"
Oct 30 23:29:17.123761 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:17.104417    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:29:17.226669 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:17.147919    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 23:29:17.242047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:17.160692    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 30 23:29:17.557994 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:17.538970    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:17.674922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:17.662152    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded"
Oct 30 23:29:17.674922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:17.662017    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 23:29:17.854695 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:17.731916    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:17.854695 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:17.800288    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:17.891030 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:17.890981    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:17.892242 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:17.892209    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 30 23:29:17.894257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:17.894228    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 23:29:17.974316 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:17.918519    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:17.974316 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:17.945826    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:17.975020 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:17.974974    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:18.149627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.146258    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:29:18.168756 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.168721    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:29:18.244018 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.243920    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:18.251484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.249067    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Oct 30 23:29:18.259808 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.259670    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:18.312623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.285897    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:18.365892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.365845    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:29:18.366127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.366104    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56" podUID="43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:18.385894 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:29:18.385766    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:29:18.451431 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.451402    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:29:18.452048 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.452028    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:29:18.463091 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.463070    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:29:18.466757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.466740    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:29:18.505611 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.505510    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww"]
Oct 30 23:29:18.601642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.598840    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww"]
Oct 30 23:29:18.633731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.606756    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b"
Oct 30 23:29:18.633731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.620588    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:18.685705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.667268    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded"
Oct 30 23:29:18.731896 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.721153    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-maas-api-openshift-ca\") pod \"75e243a4-d9b3-48c3-a5e3-a548c2ea253d\" (UID: \"75e243a4-d9b3-48c3-a5e3-a548c2ea253d\") "
Oct 30 23:29:18.731896 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.721973    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-maas-api-xks-ca\") pod \"75e243a4-d9b3-48c3-a5e3-a548c2ea253d\" (UID: \"75e243a4-d9b3-48c3-a5e3-a548c2ea253d\") "
Oct 30 23:29:18.731896 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.722211    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-kube-api-access-kkwsl\" (UniqueName: \"kubernetes.io/projected/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-kube-api-access-kkwsl\") pod \"75e243a4-d9b3-48c3-a5e3-a548c2ea253d\" (UID: \"75e243a4-d9b3-48c3-a5e3-a548c2ea253d\") "
Oct 30 23:29:18.731896 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.722438    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad52a332-c204-4c8b-b4ef-e70474665391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d149c530-7pdww\" (UID: \"ad52a332-c204-4c8b-b4ef-e70474665391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww"
Oct 30 23:29:18.731896 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.722529    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxdsl\" (UniqueName: \"kubernetes.io/projected/ad52a332-c204-4c8b-b4ef-e70474665391-kube-api-access-fxdsl\") pod \"maas-api-revoke-sub-d149c530-7pdww\" (UID: \"ad52a332-c204-4c8b-b4ef-e70474665391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww"
Oct 30 23:29:18.731896 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.728258    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad52a332-c204-4c8b-b4ef-e70474665391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d149c530-7pdww\" (UID: \"ad52a332-c204-4c8b-b4ef-e70474665391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww"
Oct 30 23:29:18.731896 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.729563    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww"
Oct 30 23:29:18.731896 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.730377    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-maas-api-openshift-ca" pod "75e243a4-d9b3-48c3-a5e3-a548c2ea253d" (UID: "75e243a4-d9b3-48c3-a5e3-a548c2ea253d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:29:18.761285 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.732947    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:18.761285 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.741792    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:29:18.761285 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.743948    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:29:18.761285 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.759836    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:29:18.849082 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.813718    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 23:29:18.877680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.851303    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad52a332-c204-4c8b-b4ef-e70474665391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d149c530-7pdww\" (UID: \"ad52a332-c204-4c8b-b4ef-e70474665391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww"
Oct 30 23:29:18.877680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.851663    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad52a332-c204-4c8b-b4ef-e70474665391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d149c530-7pdww\" (UID: \"ad52a332-c204-4c8b-b4ef-e70474665391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww"
Oct 30 23:29:18.877680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.851750    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxdsl\" (UniqueName: \"kubernetes.io/projected/ad52a332-c204-4c8b-b4ef-e70474665391-kube-api-access-fxdsl\") pod \"maas-api-revoke-sub-d149c530-7pdww\" (UID: \"ad52a332-c204-4c8b-b4ef-e70474665391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww"
Oct 30 23:29:18.877680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.857473    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:18.877680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.858482    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad52a332-c204-4c8b-b4ef-e70474665391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d149c530-7pdww\" (UID: \"ad52a332-c204-4c8b-b4ef-e70474665391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww"
Oct 30 23:29:18.892616 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.878889    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-maas-api-xks-ca" pod "75e243a4-d9b3-48c3-a5e3-a548c2ea253d" (UID: "75e243a4-d9b3-48c3-a5e3-a548c2ea253d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:29:18.892616 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.880001    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:29:18.892616 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.880567    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:29:18.920120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.915479    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-kube-api-access-kkwsl" pod "75e243a4-d9b3-48c3-a5e3-a548c2ea253d" (UID: "75e243a4-d9b3-48c3-a5e3-a548c2ea253d"). InnerVolumeSpecName "kube-api-access-kkwsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:29:18.920120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.916300    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad52a332-c204-4c8b-b4ef-e70474665391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d149c530-7pdww\" (UID: \"ad52a332-c204-4c8b-b4ef-e70474665391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww"
Oct 30 23:29:18.940426 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.940399    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:29:18.941817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.941798    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v"]
Oct 30 23:29:18.943774 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.943710    5078 cpu_manager.go:414] "containerMap: removing container" podUID="75e243a4-d9b3-48c3-a5e3-a548c2ea253d" containerName="revoke-keys"
Oct 30 23:29:18.943955 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.943940    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e243a4-d9b3-48c3-a5e3-a548c2ea253d" containerName="revoke-keys"
Oct 30 23:29:18.944310 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.944287    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e243a4-d9b3-48c3-a5e3-a548c2ea253d" containerName="revoke-keys"
Oct 30 23:29:18.960521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.953512    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxdsl\" (UniqueName: \"kubernetes.io/projected/ad52a332-c204-4c8b-b4ef-e70474665391-kube-api-access-fxdsl\") pod \"maas-api-revoke-sub-d149c530-7pdww\" (UID: \"ad52a332-c204-4c8b-b4ef-e70474665391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww"
Oct 30 23:29:18.961267 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.960861    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:18.961267 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.960886    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkwsl\" (UniqueName: \"kubernetes.io/projected/75e243a4-d9b3-48c3-a5e3-a548c2ea253d-kube-api-access-kkwsl\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:18.964478 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.964452    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:29:18.964816 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.964799    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v"
Oct 30 23:29:18.966582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.966548    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v"]
Oct 30 23:29:18.979866 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:18.979847    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww"
Oct 30 23:29:19.062098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:19.062067    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-237d62a6-xkn6v\" (UID: \"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v"
Oct 30 23:29:19.062291 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:19.062274    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-237d62a6-xkn6v\" (UID: \"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v"
Oct 30 23:29:19.062541 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:19.062516    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tssrd\" (UniqueName: \"kubernetes.io/projected/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-kube-api-access-tssrd\") pod \"maas-api-revoke-sub-237d62a6-xkn6v\" (UID: \"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v"
Oct 30 23:29:19.163403 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:19.163365    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-237d62a6-xkn6v\" (UID: \"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v"
Oct 30 23:29:19.163619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:19.163438    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tssrd\" (UniqueName: \"kubernetes.io/projected/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-kube-api-access-tssrd\") pod \"maas-api-revoke-sub-237d62a6-xkn6v\" (UID: \"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v"
Oct 30 23:29:19.163619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:19.163586    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-237d62a6-xkn6v\" (UID: \"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v"
Oct 30 23:29:19.164181 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:19.164160    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-237d62a6-xkn6v\" (UID: \"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v"
Oct 30 23:29:19.165867 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:19.165846    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-maas-api-xks-ca\") pod \"maas-api-revoke-sub-237d62a6-xkn6v\" (UID: \"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v"
Oct 30 23:29:19.219908 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:19.219874    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tssrd\" (UniqueName: \"kubernetes.io/projected/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-kube-api-access-tssrd\") pod \"maas-api-revoke-sub-237d62a6-xkn6v\" (UID: \"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v"
Oct 30 23:29:19.233990 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:29:19.233959    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1f15a7_80d1_469a_ad5d_b06fd58aaf3a.slice/crio-27702f2ebd18c2cb5fdcd55455e8e07a1e55fe02cc842dd0a6fbf43a036295b3 WatchSource:0}: Error finding container 27702f2ebd18c2cb5fdcd55455e8e07a1e55fe02cc842dd0a6fbf43a036295b3: Status 404 returned error can't find the container with id 27702f2ebd18c2cb5fdcd55455e8e07a1e55fe02cc842dd0a6fbf43a036295b3
Oct 30 23:29:19.235556 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:19.235529    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942"]
Oct 30 23:29:19.293066 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:19.293034    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v"
Oct 30 23:29:19.319153 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:19.319127    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b"
Oct 30 23:29:19.319153 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:19.319137    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b" event={"ID":"75e243a4-d9b3-48c3-a5e3-a548c2ea253d","Type":"ContainerDied","Data":"1db60b716db59832b94b4bf729f2907962c61589874fe978f12488004e7084b8"}
Oct 30 23:29:19.319313 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:19.319191    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db60b716db59832b94b4bf729f2907962c61589874fe978f12488004e7084b8"
Oct 30 23:29:19.320673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:19.320647    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942" event={"ID":"0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a","Type":"ContainerStarted","Data":"27702f2ebd18c2cb5fdcd55455e8e07a1e55fe02cc842dd0a6fbf43a036295b3"}
Oct 30 23:29:19.378158 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:29:19.378130    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad52a332_c204_4c8b_b4ef_e70474665391.slice/crio-1f751519b4c569cfe0cba41f9099099cc719fd005fbb52ed5979db9529fa9335 WatchSource:0}: Error finding container 1f751519b4c569cfe0cba41f9099099cc719fd005fbb52ed5979db9529fa9335: Status 404 returned error can't find the container with id 1f751519b4c569cfe0cba41f9099099cc719fd005fbb52ed5979db9529fa9335
Oct 30 23:29:19.379845 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:19.379817    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww"]
Oct 30 23:29:19.702881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:19.702845    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v"]
Oct 30 23:29:20.334380 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:20.334346    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww" event={"ID":"ad52a332-c204-4c8b-b4ef-e70474665391","Type":"ContainerStarted","Data":"e4b98e17f546d49aa4db0eec3a2166038b8bc3042a8250b3d608e5ce4a736a4c"}
Oct 30 23:29:20.334649 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:20.334410    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww" event={"ID":"ad52a332-c204-4c8b-b4ef-e70474665391","Type":"ContainerStarted","Data":"1f751519b4c569cfe0cba41f9099099cc719fd005fbb52ed5979db9529fa9335"}
Oct 30 23:29:20.337277 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:20.337246    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v" event={"ID":"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309","Type":"ContainerStarted","Data":"761ccb188dc52e99a71f16b9768a034642cb215dad7d282fa767d625ed5ea6e6"}
Oct 30 23:29:20.337371 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:20.337300    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v" event={"ID":"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309","Type":"ContainerStarted","Data":"8f0e58af7f0f5bce5b9e02e679ddf9e5d17ed50baf6d57ec58a2b0b9d5fc7732"}
Oct 30 23:29:20.339179 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:20.339153    5078 generic.go:356] "Generic (PLEG): container finished" podID="0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a" containerID="166731348a496c6251b00645ad8de451e8386c0bbe43af67c02a23e9c02cac9e" exitCode=0
Oct 30 23:29:20.339279 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:20.339249    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942" event={"ID":"0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a","Type":"ContainerDied","Data":"166731348a496c6251b00645ad8de451e8386c0bbe43af67c02a23e9c02cac9e"}
Oct 30 23:29:20.917976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:20.917918    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww" podStartSLOduration=4.917900673 podStartE2EDuration="4.917900673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:29:20.915353828 +0000 UTC m=+1346.803260186" watchObservedRunningTime="2026-10-30 23:29:20.917900673 +0000 UTC m=+1346.805807029"
Oct 30 23:29:21.147416 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:21.147353    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v" podStartSLOduration=3.147339455 podStartE2EDuration="3.147339455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:29:21.142989 +0000 UTC m=+1347.030895353" watchObservedRunningTime="2026-10-30 23:29:21.147339455 +0000 UTC m=+1347.035245856"
Oct 30 23:29:21.348012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:21.347979    5078 generic.go:356] "Generic (PLEG): container finished" podID="ad52a332-c204-4c8b-b4ef-e70474665391" containerID="e4b98e17f546d49aa4db0eec3a2166038b8bc3042a8250b3d608e5ce4a736a4c" exitCode=0
Oct 30 23:29:21.348171 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:21.348069    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww" event={"ID":"ad52a332-c204-4c8b-b4ef-e70474665391","Type":"ContainerDied","Data":"e4b98e17f546d49aa4db0eec3a2166038b8bc3042a8250b3d608e5ce4a736a4c"}
Oct 30 23:29:21.349650 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:21.349629    5078 generic.go:356] "Generic (PLEG): container finished" podID="15eb2805-3deb-43f1-bdd9-8cb3c9dc3309" containerID="761ccb188dc52e99a71f16b9768a034642cb215dad7d282fa767d625ed5ea6e6" exitCode=0
Oct 30 23:29:21.349771 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:21.349655    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v" event={"ID":"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309","Type":"ContainerDied","Data":"761ccb188dc52e99a71f16b9768a034642cb215dad7d282fa767d625ed5ea6e6"}
Oct 30 23:29:21.498718 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:21.498698    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942"
Oct 30 23:29:21.633796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:21.633770    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-kube-api-access-n68fp\" (UniqueName: \"kubernetes.io/projected/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-kube-api-access-n68fp\") pod \"0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a\" (UID: \"0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a\") "
Oct 30 23:29:21.633952 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:21.633911    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-maas-api-openshift-ca\") pod \"0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a\" (UID: \"0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a\") "
Oct 30 23:29:21.634002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:21.633977    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-maas-api-xks-ca\") pod \"0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a\" (UID: \"0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a\") "
Oct 30 23:29:21.634298 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:21.634267    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-maas-api-openshift-ca" pod "0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a" (UID: "0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:29:21.634642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:21.634522    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:21.635996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:21.635969    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-kube-api-access-n68fp" pod "0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a" (UID: "0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a"). InnerVolumeSpecName "kube-api-access-n68fp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:29:21.636089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:21.636037    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-maas-api-xks-ca" pod "0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a" (UID: "0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:29:21.735789 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:21.735712    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n68fp\" (UniqueName: \"kubernetes.io/projected/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-kube-api-access-n68fp\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:21.735789 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:21.735738    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:22.358341 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.358317    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942"
Oct 30 23:29:22.358792 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.358338    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942" event={"ID":"0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a","Type":"ContainerDied","Data":"27702f2ebd18c2cb5fdcd55455e8e07a1e55fe02cc842dd0a6fbf43a036295b3"}
Oct 30 23:29:22.358792 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.358387    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27702f2ebd18c2cb5fdcd55455e8e07a1e55fe02cc842dd0a6fbf43a036295b3"
Oct 30 23:29:22.576879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.576853    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww"
Oct 30 23:29:22.579274 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.579257    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v"
Oct 30 23:29:22.650140 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.650118    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad52a332-c204-4c8b-b4ef-e70474665391-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad52a332-c204-4c8b-b4ef-e70474665391-maas-api-xks-ca\") pod \"ad52a332-c204-4c8b-b4ef-e70474665391\" (UID: \"ad52a332-c204-4c8b-b4ef-e70474665391\") "
Oct 30 23:29:22.650283 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.650206    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-kube-api-access-tssrd\" (UniqueName: \"kubernetes.io/projected/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-kube-api-access-tssrd\") pod \"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309\" (UID: \"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309\") "
Oct 30 23:29:22.650283 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.650247    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad52a332-c204-4c8b-b4ef-e70474665391-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad52a332-c204-4c8b-b4ef-e70474665391-maas-api-openshift-ca\") pod \"ad52a332-c204-4c8b-b4ef-e70474665391\" (UID: \"ad52a332-c204-4c8b-b4ef-e70474665391\") "
Oct 30 23:29:22.650407 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.650294    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-maas-api-openshift-ca\") pod \"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309\" (UID: \"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309\") "
Oct 30 23:29:22.650407 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.650379    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-maas-api-xks-ca\") pod \"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309\" (UID: \"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309\") "
Oct 30 23:29:22.650521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.650431    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad52a332-c204-4c8b-b4ef-e70474665391-kube-api-access-fxdsl\" (UniqueName: \"kubernetes.io/projected/ad52a332-c204-4c8b-b4ef-e70474665391-kube-api-access-fxdsl\") pod \"ad52a332-c204-4c8b-b4ef-e70474665391\" (UID: \"ad52a332-c204-4c8b-b4ef-e70474665391\") "
Oct 30 23:29:22.650717 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.650657    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad52a332-c204-4c8b-b4ef-e70474665391-maas-api-openshift-ca" pod "ad52a332-c204-4c8b-b4ef-e70474665391" (UID: "ad52a332-c204-4c8b-b4ef-e70474665391"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:29:22.650717 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.650689    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-maas-api-openshift-ca" pod "15eb2805-3deb-43f1-bdd9-8cb3c9dc3309" (UID: "15eb2805-3deb-43f1-bdd9-8cb3c9dc3309"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:29:22.651074 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.651013    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:22.651074 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.651036    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad52a332-c204-4c8b-b4ef-e70474665391-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:22.652666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.652641    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-kube-api-access-tssrd" pod "15eb2805-3deb-43f1-bdd9-8cb3c9dc3309" (UID: "15eb2805-3deb-43f1-bdd9-8cb3c9dc3309"). InnerVolumeSpecName "kube-api-access-tssrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:29:22.652756 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.652718    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-maas-api-xks-ca" pod "15eb2805-3deb-43f1-bdd9-8cb3c9dc3309" (UID: "15eb2805-3deb-43f1-bdd9-8cb3c9dc3309"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:29:22.652920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.652893    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad52a332-c204-4c8b-b4ef-e70474665391-maas-api-xks-ca" pod "ad52a332-c204-4c8b-b4ef-e70474665391" (UID: "ad52a332-c204-4c8b-b4ef-e70474665391"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:29:22.653022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.652992    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad52a332-c204-4c8b-b4ef-e70474665391-kube-api-access-fxdsl" pod "ad52a332-c204-4c8b-b4ef-e70474665391" (UID: "ad52a332-c204-4c8b-b4ef-e70474665391"). InnerVolumeSpecName "kube-api-access-fxdsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:29:22.752423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.752381    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tssrd\" (UniqueName: \"kubernetes.io/projected/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-kube-api-access-tssrd\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:22.752423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.752419    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:22.752423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.752429    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxdsl\" (UniqueName: \"kubernetes.io/projected/ad52a332-c204-4c8b-b4ef-e70474665391-kube-api-access-fxdsl\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:22.752423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:22.752438    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad52a332-c204-4c8b-b4ef-e70474665391-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:23.367549 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.367519    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww"
Oct 30 23:29:23.368069 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.367559    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww" event={"ID":"ad52a332-c204-4c8b-b4ef-e70474665391","Type":"ContainerDied","Data":"1f751519b4c569cfe0cba41f9099099cc719fd005fbb52ed5979db9529fa9335"}
Oct 30 23:29:23.368069 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.367642    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f751519b4c569cfe0cba41f9099099cc719fd005fbb52ed5979db9529fa9335"
Oct 30 23:29:23.369047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.369026    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v"
Oct 30 23:29:23.369178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.369055    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v" event={"ID":"15eb2805-3deb-43f1-bdd9-8cb3c9dc3309","Type":"ContainerDied","Data":"8f0e58af7f0f5bce5b9e02e679ddf9e5d17ed50baf6d57ec58a2b0b9d5fc7732"}
Oct 30 23:29:23.369178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.369099    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f0e58af7f0f5bce5b9e02e679ddf9e5d17ed50baf6d57ec58a2b0b9d5fc7732"
Oct 30 23:29:23.583231 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.583201    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m"]
Oct 30 23:29:23.583829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.583813    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ad52a332-c204-4c8b-b4ef-e70474665391" containerName="revoke-keys"
Oct 30 23:29:23.583902 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.583830    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad52a332-c204-4c8b-b4ef-e70474665391" containerName="revoke-keys"
Oct 30 23:29:23.583902 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.583853    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a" containerName="revoke-keys"
Oct 30 23:29:23.583902 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.583858    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a" containerName="revoke-keys"
Oct 30 23:29:23.583902 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.583871    5078 cpu_manager.go:414] "containerMap: removing container" podUID="15eb2805-3deb-43f1-bdd9-8cb3c9dc3309" containerName="revoke-keys"
Oct 30 23:29:23.583902 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.583876    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15eb2805-3deb-43f1-bdd9-8cb3c9dc3309" containerName="revoke-keys"
Oct 30 23:29:23.584055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.583986    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a" containerName="revoke-keys"
Oct 30 23:29:23.584055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.584001    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="15eb2805-3deb-43f1-bdd9-8cb3c9dc3309" containerName="revoke-keys"
Oct 30 23:29:23.584055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.584016    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad52a332-c204-4c8b-b4ef-e70474665391" containerName="revoke-keys"
Oct 30 23:29:23.587713 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.587697    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m"
Oct 30 23:29:23.642664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.642562    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:29:23.642924 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.642904    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:29:23.643116 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.643092    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:29:23.643589 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.643236    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:29:23.645360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.645333    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m"]
Oct 30 23:29:23.663044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.663020    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25027913-wm67m\" (UID: \"31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m"
Oct 30 23:29:23.663187 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.663169    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwh9j\" (UniqueName: \"kubernetes.io/projected/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-kube-api-access-kwh9j\") pod \"maas-api-revoke-sub-25027913-wm67m\" (UID: \"31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m"
Oct 30 23:29:23.663333 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.663308    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25027913-wm67m\" (UID: \"31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m"
Oct 30 23:29:23.764913 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.764884    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25027913-wm67m\" (UID: \"31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m"
Oct 30 23:29:23.765061 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.764952    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwh9j\" (UniqueName: \"kubernetes.io/projected/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-kube-api-access-kwh9j\") pod \"maas-api-revoke-sub-25027913-wm67m\" (UID: \"31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m"
Oct 30 23:29:23.765061 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.765028    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25027913-wm67m\" (UID: \"31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m"
Oct 30 23:29:23.765624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.765571    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25027913-wm67m\" (UID: \"31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m"
Oct 30 23:29:23.767480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.767458    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25027913-wm67m\" (UID: \"31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m"
Oct 30 23:29:23.822939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.822919    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwh9j\" (UniqueName: \"kubernetes.io/projected/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-kube-api-access-kwh9j\") pod \"maas-api-revoke-sub-25027913-wm67m\" (UID: \"31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m"
Oct 30 23:29:23.902833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:23.902756    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m"
Oct 30 23:29:24.282645 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:29:24.282584    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d4d0c1_bda3_494f_86f4_1a6f2dd4fd8f.slice/crio-5fec81cc04b7e2949eef9d43f77877b1e59d7659862b06b9bd122b0d6d0d5a66 WatchSource:0}: Error finding container 5fec81cc04b7e2949eef9d43f77877b1e59d7659862b06b9bd122b0d6d0d5a66: Status 404 returned error can't find the container with id 5fec81cc04b7e2949eef9d43f77877b1e59d7659862b06b9bd122b0d6d0d5a66
Oct 30 23:29:24.283587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:24.283555    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m"]
Oct 30 23:29:24.378816 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:24.378790    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m" event={"ID":"31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f","Type":"ContainerStarted","Data":"5fec81cc04b7e2949eef9d43f77877b1e59d7659862b06b9bd122b0d6d0d5a66"}
Oct 30 23:29:25.047516 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:25.047484    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw"]
Oct 30 23:29:25.096619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:25.096576    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35aede95-8t9sw"]
Oct 30 23:29:25.388054 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:25.387974    5078 generic.go:356] "Generic (PLEG): container finished" podID="31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f" containerID="ceb6b951397bb18b3771a576a7ce94c7ef5174458ecc095744ec2de072400afd" exitCode=0
Oct 30 23:29:25.388474 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:25.388091    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m" event={"ID":"31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f","Type":"ContainerDied","Data":"ceb6b951397bb18b3771a576a7ce94c7ef5174458ecc095744ec2de072400afd"}
Oct 30 23:29:26.299290 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:26.299254    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e1d2e37-4880-49ae-85ad-b24b06578beb" path="/var/lib/kubelet/pods/4e1d2e37-4880-49ae-85ad-b24b06578beb/volumes"
Oct 30 23:29:26.550784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:26.550726    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m"
Oct 30 23:29:26.731620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:26.725560    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-maas-api-openshift-ca\") pod \"31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f\" (UID: \"31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f\") "
Oct 30 23:29:26.731620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:26.725853    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-kube-api-access-kwh9j\" (UniqueName: \"kubernetes.io/projected/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-kube-api-access-kwh9j\") pod \"31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f\" (UID: \"31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f\") "
Oct 30 23:29:26.731620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:26.726133    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-maas-api-xks-ca\") pod \"31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f\" (UID: \"31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f\") "
Oct 30 23:29:26.731620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:26.731417    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-maas-api-openshift-ca" pod "31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f" (UID: "31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:29:26.774345 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:26.774288    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-maas-api-xks-ca" pod "31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f" (UID: "31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:29:26.787731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:26.787403    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-kube-api-access-kwh9j" pod "31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f" (UID: "31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f"). InnerVolumeSpecName "kube-api-access-kwh9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:29:26.828330 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:26.828248    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:26.828330 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:26.828276    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:26.828330 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:26.828293    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwh9j\" (UniqueName: \"kubernetes.io/projected/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f-kube-api-access-kwh9j\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:27.221071 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.220994    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26"]
Oct 30 23:29:27.221861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.221833    5078 cpu_manager.go:414] "containerMap: removing container" podUID="31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f" containerName="revoke-keys"
Oct 30 23:29:27.221861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.221863    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f" containerName="revoke-keys"
Oct 30 23:29:27.222022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.222011    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f" containerName="revoke-keys"
Oct 30 23:29:27.226148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.226130    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26"
Oct 30 23:29:27.277185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.277151    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26"]
Oct 30 23:29:27.334434 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.334399    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72cedd5d-767a-4e31-855f-0578981b8c3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3df90abd-bkw26\" (UID: \"72cedd5d-767a-4e31-855f-0578981b8c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26"
Oct 30 23:29:27.334617 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.334547    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72cedd5d-767a-4e31-855f-0578981b8c3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3df90abd-bkw26\" (UID: \"72cedd5d-767a-4e31-855f-0578981b8c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26"
Oct 30 23:29:27.334659 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.334589    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jc4f\" (UniqueName: \"kubernetes.io/projected/72cedd5d-767a-4e31-855f-0578981b8c3b-kube-api-access-7jc4f\") pod \"maas-api-revoke-sub-3df90abd-bkw26\" (UID: \"72cedd5d-767a-4e31-855f-0578981b8c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26"
Oct 30 23:29:27.405235 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.405208    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m" event={"ID":"31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f","Type":"ContainerDied","Data":"5fec81cc04b7e2949eef9d43f77877b1e59d7659862b06b9bd122b0d6d0d5a66"}
Oct 30 23:29:27.405235 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.405233    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m"
Oct 30 23:29:27.405423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.405251    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fec81cc04b7e2949eef9d43f77877b1e59d7659862b06b9bd122b0d6d0d5a66"
Oct 30 23:29:27.436047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.436022    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72cedd5d-767a-4e31-855f-0578981b8c3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3df90abd-bkw26\" (UID: \"72cedd5d-767a-4e31-855f-0578981b8c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26"
Oct 30 23:29:27.436178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.436165    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72cedd5d-767a-4e31-855f-0578981b8c3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3df90abd-bkw26\" (UID: \"72cedd5d-767a-4e31-855f-0578981b8c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26"
Oct 30 23:29:27.436246 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.436234    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jc4f\" (UniqueName: \"kubernetes.io/projected/72cedd5d-767a-4e31-855f-0578981b8c3b-kube-api-access-7jc4f\") pod \"maas-api-revoke-sub-3df90abd-bkw26\" (UID: \"72cedd5d-767a-4e31-855f-0578981b8c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26"
Oct 30 23:29:27.436936 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.436850    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72cedd5d-767a-4e31-855f-0578981b8c3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3df90abd-bkw26\" (UID: \"72cedd5d-767a-4e31-855f-0578981b8c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26"
Oct 30 23:29:27.438577 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.438556    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72cedd5d-767a-4e31-855f-0578981b8c3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3df90abd-bkw26\" (UID: \"72cedd5d-767a-4e31-855f-0578981b8c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26"
Oct 30 23:29:27.494221 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.494158    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jc4f\" (UniqueName: \"kubernetes.io/projected/72cedd5d-767a-4e31-855f-0578981b8c3b-kube-api-access-7jc4f\") pod \"maas-api-revoke-sub-3df90abd-bkw26\" (UID: \"72cedd5d-767a-4e31-855f-0578981b8c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26"
Oct 30 23:29:27.539991 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.539962    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26"
Oct 30 23:29:27.918626 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:29:27.918577    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72cedd5d_767a_4e31_855f_0578981b8c3b.slice/crio-2ad8f1c91417d3cf51b515589f1bd2e6e06798f1fe988872bba256330c6be929 WatchSource:0}: Error finding container 2ad8f1c91417d3cf51b515589f1bd2e6e06798f1fe988872bba256330c6be929: Status 404 returned error can't find the container with id 2ad8f1c91417d3cf51b515589f1bd2e6e06798f1fe988872bba256330c6be929
Oct 30 23:29:27.919055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:27.919029    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26"]
Oct 30 23:29:28.421185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:28.421095    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26" event={"ID":"72cedd5d-767a-4e31-855f-0578981b8c3b","Type":"ContainerStarted","Data":"a1fa1868ba803443e529227ad1cca3c89819546af8b179f861de2e94acfb8415"}
Oct 30 23:29:28.421335 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:28.421190    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26" event={"ID":"72cedd5d-767a-4e31-855f-0578981b8c3b","Type":"ContainerStarted","Data":"2ad8f1c91417d3cf51b515589f1bd2e6e06798f1fe988872bba256330c6be929"}
Oct 30 23:29:28.535607 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:28.535541    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26" podStartSLOduration=1.535525733 podStartE2EDuration="1.535525733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:29:28.535066306 +0000 UTC m=+1354.422972659" watchObservedRunningTime="2026-10-30 23:29:28.535525733 +0000 UTC m=+1354.423432166"
Oct 30 23:29:29.553867 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:29.553833    5078 generic.go:356] "Generic (PLEG): container finished" podID="72cedd5d-767a-4e31-855f-0578981b8c3b" containerID="a1fa1868ba803443e529227ad1cca3c89819546af8b179f861de2e94acfb8415" exitCode=0
Oct 30 23:29:29.554316 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:29.554018    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26" event={"ID":"72cedd5d-767a-4e31-855f-0578981b8c3b","Type":"ContainerDied","Data":"a1fa1868ba803443e529227ad1cca3c89819546af8b179f861de2e94acfb8415"}
Oct 30 23:29:30.825480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:30.825454    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26"
Oct 30 23:29:30.888240 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:30.888163    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72cedd5d-767a-4e31-855f-0578981b8c3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72cedd5d-767a-4e31-855f-0578981b8c3b-maas-api-xks-ca\") pod \"72cedd5d-767a-4e31-855f-0578981b8c3b\" (UID: \"72cedd5d-767a-4e31-855f-0578981b8c3b\") "
Oct 30 23:29:30.888394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:30.888343    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72cedd5d-767a-4e31-855f-0578981b8c3b-kube-api-access-7jc4f\" (UniqueName: \"kubernetes.io/projected/72cedd5d-767a-4e31-855f-0578981b8c3b-kube-api-access-7jc4f\") pod \"72cedd5d-767a-4e31-855f-0578981b8c3b\" (UID: \"72cedd5d-767a-4e31-855f-0578981b8c3b\") "
Oct 30 23:29:30.888455 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:30.888430    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72cedd5d-767a-4e31-855f-0578981b8c3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72cedd5d-767a-4e31-855f-0578981b8c3b-maas-api-openshift-ca\") pod \"72cedd5d-767a-4e31-855f-0578981b8c3b\" (UID: \"72cedd5d-767a-4e31-855f-0578981b8c3b\") "
Oct 30 23:29:30.888806 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:30.888781    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72cedd5d-767a-4e31-855f-0578981b8c3b-maas-api-openshift-ca" pod "72cedd5d-767a-4e31-855f-0578981b8c3b" (UID: "72cedd5d-767a-4e31-855f-0578981b8c3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:29:30.888965 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:30.888950    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72cedd5d-767a-4e31-855f-0578981b8c3b-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:30.890391 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:30.890365    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72cedd5d-767a-4e31-855f-0578981b8c3b-maas-api-xks-ca" pod "72cedd5d-767a-4e31-855f-0578981b8c3b" (UID: "72cedd5d-767a-4e31-855f-0578981b8c3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:29:30.890565 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:30.890542    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72cedd5d-767a-4e31-855f-0578981b8c3b-kube-api-access-7jc4f" pod "72cedd5d-767a-4e31-855f-0578981b8c3b" (UID: "72cedd5d-767a-4e31-855f-0578981b8c3b"). InnerVolumeSpecName "kube-api-access-7jc4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:29:30.990779 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:30.990750    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jc4f\" (UniqueName: \"kubernetes.io/projected/72cedd5d-767a-4e31-855f-0578981b8c3b-kube-api-access-7jc4f\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:30.990779 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:30.990777    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72cedd5d-767a-4e31-855f-0578981b8c3b-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:31.633288 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:31.633261    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26"
Oct 30 23:29:31.633454 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:31.633294    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26" event={"ID":"72cedd5d-767a-4e31-855f-0578981b8c3b","Type":"ContainerDied","Data":"2ad8f1c91417d3cf51b515589f1bd2e6e06798f1fe988872bba256330c6be929"}
Oct 30 23:29:31.633454 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:31.633372    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ad8f1c91417d3cf51b515589f1bd2e6e06798f1fe988872bba256330c6be929"
Oct 30 23:29:33.921738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:33.921660    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx"]
Oct 30 23:29:33.922520 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:33.922503    5078 cpu_manager.go:414] "containerMap: removing container" podUID="72cedd5d-767a-4e31-855f-0578981b8c3b" containerName="revoke-keys"
Oct 30 23:29:33.922636 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:33.922522    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72cedd5d-767a-4e31-855f-0578981b8c3b" containerName="revoke-keys"
Oct 30 23:29:33.922697 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:33.922685    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="72cedd5d-767a-4e31-855f-0578981b8c3b" containerName="revoke-keys"
Oct 30 23:29:33.927952 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:33.927933    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx"
Oct 30 23:29:33.984878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:33.984847    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx"]
Oct 30 23:29:34.028216 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:34.028185    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:29:34.028216 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:34.028196    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:29:34.032521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:34.032505    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:29:34.032731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:34.032715    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:29:34.037487 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:34.037465    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29ec09e1-96mfx\" (UID: \"3ab1f8c9-cc86-4e48-b679-d38e1e6ad418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx"
Oct 30 23:29:34.037547 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:34.037501    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gvg6\" (UniqueName: \"kubernetes.io/projected/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-kube-api-access-4gvg6\") pod \"maas-api-revoke-sub-29ec09e1-96mfx\" (UID: \"3ab1f8c9-cc86-4e48-b679-d38e1e6ad418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx"
Oct 30 23:29:34.037653 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:34.037639    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29ec09e1-96mfx\" (UID: \"3ab1f8c9-cc86-4e48-b679-d38e1e6ad418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx"
Oct 30 23:29:34.138888 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:34.138857    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29ec09e1-96mfx\" (UID: \"3ab1f8c9-cc86-4e48-b679-d38e1e6ad418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx"
Oct 30 23:29:34.139081 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:34.138894    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gvg6\" (UniqueName: \"kubernetes.io/projected/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-kube-api-access-4gvg6\") pod \"maas-api-revoke-sub-29ec09e1-96mfx\" (UID: \"3ab1f8c9-cc86-4e48-b679-d38e1e6ad418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx"
Oct 30 23:29:34.139081 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:34.139028    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29ec09e1-96mfx\" (UID: \"3ab1f8c9-cc86-4e48-b679-d38e1e6ad418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx"
Oct 30 23:29:34.139539 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:34.139509    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29ec09e1-96mfx\" (UID: \"3ab1f8c9-cc86-4e48-b679-d38e1e6ad418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx"
Oct 30 23:29:34.141462 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:34.141440    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29ec09e1-96mfx\" (UID: \"3ab1f8c9-cc86-4e48-b679-d38e1e6ad418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx"
Oct 30 23:29:34.196366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:34.196292    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gvg6\" (UniqueName: \"kubernetes.io/projected/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-kube-api-access-4gvg6\") pod \"maas-api-revoke-sub-29ec09e1-96mfx\" (UID: \"3ab1f8c9-cc86-4e48-b679-d38e1e6ad418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx"
Oct 30 23:29:34.242300 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:34.242274    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx"
Oct 30 23:29:34.635738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:34.635710    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx"]
Oct 30 23:29:34.636693 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:29:34.636666    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab1f8c9_cc86_4e48_b679_d38e1e6ad418.slice/crio-33a2d3ca8cd05e059419fa01b36fba845065c0a60935a8e239998f61365bd1f0 WatchSource:0}: Error finding container 33a2d3ca8cd05e059419fa01b36fba845065c0a60935a8e239998f61365bd1f0: Status 404 returned error can't find the container with id 33a2d3ca8cd05e059419fa01b36fba845065c0a60935a8e239998f61365bd1f0
Oct 30 23:29:34.660282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:34.660256    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx" event={"ID":"3ab1f8c9-cc86-4e48-b679-d38e1e6ad418","Type":"ContainerStarted","Data":"33a2d3ca8cd05e059419fa01b36fba845065c0a60935a8e239998f61365bd1f0"}
Oct 30 23:29:35.670764 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:35.670730    5078 generic.go:356] "Generic (PLEG): container finished" podID="3ab1f8c9-cc86-4e48-b679-d38e1e6ad418" containerID="6d2c4db6a5c435e90da35bf26c2a58fa44a99ac56992d2d4946a6f79d0cdedc2" exitCode=0
Oct 30 23:29:35.671092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:35.670809    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx" event={"ID":"3ab1f8c9-cc86-4e48-b679-d38e1e6ad418","Type":"ContainerDied","Data":"6d2c4db6a5c435e90da35bf26c2a58fa44a99ac56992d2d4946a6f79d0cdedc2"}
Oct 30 23:29:36.124731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:36.124694    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s"]
Oct 30 23:29:36.215668 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:36.204966    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4a128de-7778s"]
Oct 30 23:29:36.318669 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:36.318623    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9961412-5bdf-498c-91ff-97a6534629bb" path="/var/lib/kubelet/pods/c9961412-5bdf-498c-91ff-97a6534629bb/volumes"
Oct 30 23:29:37.122091 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:37.122065    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx"
Oct 30 23:29:37.243378 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:37.243347    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-maas-api-openshift-ca\") pod \"3ab1f8c9-cc86-4e48-b679-d38e1e6ad418\" (UID: \"3ab1f8c9-cc86-4e48-b679-d38e1e6ad418\") "
Oct 30 23:29:37.243679 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:37.243406    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-maas-api-xks-ca\") pod \"3ab1f8c9-cc86-4e48-b679-d38e1e6ad418\" (UID: \"3ab1f8c9-cc86-4e48-b679-d38e1e6ad418\") "
Oct 30 23:29:37.243679 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:37.243609    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-kube-api-access-4gvg6\" (UniqueName: \"kubernetes.io/projected/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-kube-api-access-4gvg6\") pod \"3ab1f8c9-cc86-4e48-b679-d38e1e6ad418\" (UID: \"3ab1f8c9-cc86-4e48-b679-d38e1e6ad418\") "
Oct 30 23:29:37.243855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:37.243812    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-maas-api-openshift-ca" pod "3ab1f8c9-cc86-4e48-b679-d38e1e6ad418" (UID: "3ab1f8c9-cc86-4e48-b679-d38e1e6ad418"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:29:37.244217 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:37.244194    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:37.245858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:37.245830    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-maas-api-xks-ca" pod "3ab1f8c9-cc86-4e48-b679-d38e1e6ad418" (UID: "3ab1f8c9-cc86-4e48-b679-d38e1e6ad418"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:29:37.246437 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:37.246423    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-kube-api-access-4gvg6" pod "3ab1f8c9-cc86-4e48-b679-d38e1e6ad418" (UID: "3ab1f8c9-cc86-4e48-b679-d38e1e6ad418"). InnerVolumeSpecName "kube-api-access-4gvg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:29:37.349625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:37.346441    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gvg6\" (UniqueName: \"kubernetes.io/projected/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-kube-api-access-4gvg6\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:37.349625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:37.346475    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:37.694507 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:37.694428    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx"
Oct 30 23:29:37.694507 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:37.694439    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx" event={"ID":"3ab1f8c9-cc86-4e48-b679-d38e1e6ad418","Type":"ContainerDied","Data":"33a2d3ca8cd05e059419fa01b36fba845065c0a60935a8e239998f61365bd1f0"}
Oct 30 23:29:37.694507 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:37.694506    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33a2d3ca8cd05e059419fa01b36fba845065c0a60935a8e239998f61365bd1f0"
Oct 30 23:29:40.441622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.440256    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"]
Oct 30 23:29:40.441622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.441265    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3ab1f8c9-cc86-4e48-b679-d38e1e6ad418" containerName="revoke-keys"
Oct 30 23:29:40.441622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.441286    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ab1f8c9-cc86-4e48-b679-d38e1e6ad418" containerName="revoke-keys"
Oct 30 23:29:40.441622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.441455    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ab1f8c9-cc86-4e48-b679-d38e1e6ad418" containerName="revoke-keys"
Oct 30 23:29:40.697431 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.697352    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.849621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.838663    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-workload-socket\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.849621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.838728    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-envoy\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.849621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.838762    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-credential-socket\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.849621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.838804    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-token\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.870621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.838857    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hncpl\" (UniqueName: \"kubernetes.io/projected/d511dc2f-93c4-41f7-a6a2-3def48b402b4-kube-api-access-hncpl\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.870621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.850206    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-workload-certs\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.870621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.850340    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istiod-ca-cert\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.870621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.850529    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-podinfo\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.870621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.850589    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-data\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.904588 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.904401    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-5f9ec6be-openshift-default-dockercfg-bg6gj\""
Oct 30 23:29:40.941127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.941089    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"]
Oct 30 23:29:40.952623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.952539    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-podinfo\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.952760 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.952588    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-data\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.952760 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.952692    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-workload-socket\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.952760 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.952729    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-envoy\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.952760 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.952754    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-credential-socket\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.952984 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.952794    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-token\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.952984 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.952857    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hncpl\" (UniqueName: \"kubernetes.io/projected/d511dc2f-93c4-41f7-a6a2-3def48b402b4-kube-api-access-hncpl\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.952984 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.952908    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-workload-certs\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.953141 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.953017    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istiod-ca-cert\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.953201 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.953166    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-workload-socket\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.953258 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.953218    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-credential-socket\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.953322 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.953306    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-data\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.953585 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.953562    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-workload-certs\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.953989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.953966    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istiod-ca-cert\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.955194 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.955165    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-envoy\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:40.955279 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:40.955251    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-podinfo\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:41.025806 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:41.025777    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-token\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:41.025963 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:41.025943    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hncpl\" (UniqueName: \"kubernetes.io/projected/d511dc2f-93c4-41f7-a6a2-3def48b402b4-kube-api-access-hncpl\") pod \"e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") " pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:41.232485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:41.232321    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:41.823377 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:41.823348    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"]
Oct 30 23:29:41.843403 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:29:41.843375    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd511dc2f_93c4_41f7_a6a2_3def48b402b4.slice/crio-fa98391d9763d3b9eda364ec3cbad047e00e049730a8a7d4f7a30cc7edcfe732 WatchSource:0}: Error finding container fa98391d9763d3b9eda364ec3cbad047e00e049730a8a7d4f7a30cc7edcfe732: Status 404 returned error can't find the container with id fa98391d9763d3b9eda364ec3cbad047e00e049730a8a7d4f7a30cc7edcfe732
Oct 30 23:29:41.846323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:41.846289    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:29:41.846444 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:41.846359    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:29:41.846444 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:41.846386    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:29:41.950947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:41.950909    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7" event={"ID":"d511dc2f-93c4-41f7-a6a2-3def48b402b4","Type":"ContainerStarted","Data":"2235418a3fbb2c50be88a3744a178e7edca02abd8dcb0facab138a7b08c3d9f8"}
Oct 30 23:29:41.951093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:41.951001    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7" event={"ID":"d511dc2f-93c4-41f7-a6a2-3def48b402b4","Type":"ContainerStarted","Data":"fa98391d9763d3b9eda364ec3cbad047e00e049730a8a7d4f7a30cc7edcfe732"}
Oct 30 23:29:42.098324 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:42.098214    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7" podStartSLOduration=2.098199452 podStartE2EDuration="2.098199452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:29:42.093531793 +0000 UTC m=+1367.981438145" watchObservedRunningTime="2026-10-30 23:29:42.098199452 +0000 UTC m=+1367.986105804"
Oct 30 23:29:42.237624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:42.236772    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:42.248623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:42.244522    5078 patch_prober.go:29] interesting pod/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 23:29:42.248623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:42.244625    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7" podUID="d511dc2f-93c4-41f7-a6a2-3def48b402b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 23:29:42.440168 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:42.440081    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"]
Oct 30 23:29:42.440390 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:42.440349    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m" podUID="a0a363bd-8ee8-4644-b685-e24ddfa39788" containerName="istio-proxy" containerID="cri-o://930ef0b0be8f46366e681b7d050945d226e432828f3b401aab8d790b244dc0c8" gracePeriod=30
Oct 30 23:29:42.445999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:42.445972    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"]
Oct 30 23:29:42.446183 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:42.446164    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp" podUID="84900961-7d7e-4bf0-9c40-6f3d0d8102a1" containerName="istio-proxy" containerID="cri-o://b89fe26163e55f5a2f54ec2c9934bdff2af9f6411da9f2916ae741e4bc33db41" gracePeriod=30
Oct 30 23:29:43.237435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:43.237404    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:43.966733 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:43.966705    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:43.967713 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:43.967693    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:29:44.724398 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:44.724369    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp"]
Oct 30 23:29:44.728940 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:44.728916    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp"
Oct 30 23:29:44.781727 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:44.781696    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:29:44.781889 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:44.781763    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:29:44.781889 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:44.781856    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:29:44.782004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:44.781918    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:29:44.784921 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:44.784897    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp"]
Oct 30 23:29:44.840653 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:44.840591    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq924\" (UniqueName: \"kubernetes.io/projected/528ef440-0f7c-4c42-b90f-55721ebf655b-kube-api-access-wq924\") pod \"maas-api-revoke-sub-666c817c-jh6hp\" (UID: \"528ef440-0f7c-4c42-b90f-55721ebf655b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp"
Oct 30 23:29:44.840841 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:44.840691    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/528ef440-0f7c-4c42-b90f-55721ebf655b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-666c817c-jh6hp\" (UID: \"528ef440-0f7c-4c42-b90f-55721ebf655b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp"
Oct 30 23:29:44.840841 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:44.840794    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/528ef440-0f7c-4c42-b90f-55721ebf655b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-666c817c-jh6hp\" (UID: \"528ef440-0f7c-4c42-b90f-55721ebf655b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp"
Oct 30 23:29:44.942042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:44.942010    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq924\" (UniqueName: \"kubernetes.io/projected/528ef440-0f7c-4c42-b90f-55721ebf655b-kube-api-access-wq924\") pod \"maas-api-revoke-sub-666c817c-jh6hp\" (UID: \"528ef440-0f7c-4c42-b90f-55721ebf655b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp"
Oct 30 23:29:44.942240 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:44.942074    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/528ef440-0f7c-4c42-b90f-55721ebf655b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-666c817c-jh6hp\" (UID: \"528ef440-0f7c-4c42-b90f-55721ebf655b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp"
Oct 30 23:29:44.942240 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:44.942147    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/528ef440-0f7c-4c42-b90f-55721ebf655b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-666c817c-jh6hp\" (UID: \"528ef440-0f7c-4c42-b90f-55721ebf655b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp"
Oct 30 23:29:44.942718 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:44.942693    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/528ef440-0f7c-4c42-b90f-55721ebf655b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-666c817c-jh6hp\" (UID: \"528ef440-0f7c-4c42-b90f-55721ebf655b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp"
Oct 30 23:29:44.944875 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:44.944854    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/528ef440-0f7c-4c42-b90f-55721ebf655b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-666c817c-jh6hp\" (UID: \"528ef440-0f7c-4c42-b90f-55721ebf655b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp"
Oct 30 23:29:44.999295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:44.999234    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq924\" (UniqueName: \"kubernetes.io/projected/528ef440-0f7c-4c42-b90f-55721ebf655b-kube-api-access-wq924\") pod \"maas-api-revoke-sub-666c817c-jh6hp\" (UID: \"528ef440-0f7c-4c42-b90f-55721ebf655b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp"
Oct 30 23:29:45.043144 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:45.043116    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp"
Oct 30 23:29:45.418866 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:29:45.418832    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528ef440_0f7c_4c42_b90f_55721ebf655b.slice/crio-af944d3f8f8bdd99bd5c5b86583068bcdf273b8bcfd473390850b9fc71705abb WatchSource:0}: Error finding container af944d3f8f8bdd99bd5c5b86583068bcdf273b8bcfd473390850b9fc71705abb: Status 404 returned error can't find the container with id af944d3f8f8bdd99bd5c5b86583068bcdf273b8bcfd473390850b9fc71705abb
Oct 30 23:29:45.419467 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:45.419439    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp"]
Oct 30 23:29:45.983831 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:45.983740    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp" event={"ID":"528ef440-0f7c-4c42-b90f-55721ebf655b","Type":"ContainerStarted","Data":"8f71cc26fd13dffad980af1693d0dbcd346263b40ed6e98c5fe6a466a489c007"}
Oct 30 23:29:45.983831 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:45.983794    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp" event={"ID":"528ef440-0f7c-4c42-b90f-55721ebf655b","Type":"ContainerStarted","Data":"af944d3f8f8bdd99bd5c5b86583068bcdf273b8bcfd473390850b9fc71705abb"}
Oct 30 23:29:46.099798 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:46.099739    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp" podStartSLOduration=2.099719485 podStartE2EDuration="2.099719485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:29:46.099003476 +0000 UTC m=+1371.986909831" watchObservedRunningTime="2026-10-30 23:29:46.099719485 +0000 UTC m=+1371.987625839"
Oct 30 23:29:46.946804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:46.946776    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm"]
Oct 30 23:29:46.952878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:46.952847    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm"
Oct 30 23:29:46.993976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:46.993931    5078 generic.go:356] "Generic (PLEG): container finished" podID="528ef440-0f7c-4c42-b90f-55721ebf655b" containerID="8f71cc26fd13dffad980af1693d0dbcd346263b40ed6e98c5fe6a466a489c007" exitCode=0
Oct 30 23:29:46.994365 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:46.994020    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp" event={"ID":"528ef440-0f7c-4c42-b90f-55721ebf655b","Type":"ContainerDied","Data":"8f71cc26fd13dffad980af1693d0dbcd346263b40ed6e98c5fe6a466a489c007"}
Oct 30 23:29:47.005797 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.005772    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-49d07c83\""
Oct 30 23:29:47.008338 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.008315    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm"]
Oct 30 23:29:47.082621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.079089    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-49d07c83-dockercfg-sgnw9\""
Oct 30 23:29:47.228666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.223351    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kk8c\" (UniqueName: \"kubernetes.io/projected/4eebd647-3459-494d-b28c-cb27866e4551-kube-api-access-9kk8c\") pod \"payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm\" (UID: \"4eebd647-3459-494d-b28c-cb27866e4551\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm"
Oct 30 23:29:47.228666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.223483    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eebd647-3459-494d-b28c-cb27866e4551-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm\" (UID: \"4eebd647-3459-494d-b28c-cb27866e4551\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm"
Oct 30 23:29:47.332224 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.330268    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kk8c\" (UniqueName: \"kubernetes.io/projected/4eebd647-3459-494d-b28c-cb27866e4551-kube-api-access-9kk8c\") pod \"payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm\" (UID: \"4eebd647-3459-494d-b28c-cb27866e4551\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm"
Oct 30 23:29:47.332224 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.330449    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eebd647-3459-494d-b28c-cb27866e4551-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm\" (UID: \"4eebd647-3459-494d-b28c-cb27866e4551\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm"
Oct 30 23:29:47.332224 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.331485    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eebd647-3459-494d-b28c-cb27866e4551-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm\" (UID: \"4eebd647-3459-494d-b28c-cb27866e4551\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm"
Oct 30 23:29:47.421038 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.420994    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kk8c\" (UniqueName: \"kubernetes.io/projected/4eebd647-3459-494d-b28c-cb27866e4551-kube-api-access-9kk8c\") pod \"payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm\" (UID: \"4eebd647-3459-494d-b28c-cb27866e4551\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm"
Oct 30 23:29:47.572967 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.572940    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm"
Oct 30 23:29:47.600490 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:29:47.600450    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:29:47.737650 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.737565    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:29:47.778669 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.778645    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:29:47.837011 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.836983    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-podinfo\") pod \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") "
Oct 30 23:29:47.837187 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837024    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0a363bd-8ee8-4644-b685-e24ddfa39788-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a363bd-8ee8-4644-b685-e24ddfa39788-istiod-ca-cert\") pod \"a0a363bd-8ee8-4644-b685-e24ddfa39788\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") "
Oct 30 23:29:47.837187 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837051    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-credential-socket\") pod \"a0a363bd-8ee8-4644-b685-e24ddfa39788\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") "
Oct 30 23:29:47.837187 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837074    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-credential-socket\") pod \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") "
Oct 30 23:29:47.837187 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837111    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-kube-api-access-vz9vn\" (UniqueName: \"kubernetes.io/projected/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-kube-api-access-vz9vn\") pod \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") "
Oct 30 23:29:47.837187 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837131    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-data\") pod \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") "
Oct 30 23:29:47.837187 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837159    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a363bd-8ee8-4644-b685-e24ddfa39788-kube-api-access-ttmwf\" (UniqueName: \"kubernetes.io/projected/a0a363bd-8ee8-4644-b685-e24ddfa39788-kube-api-access-ttmwf\") pod \"a0a363bd-8ee8-4644-b685-e24ddfa39788\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") "
Oct 30 23:29:47.837512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837261    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-workload-certs\") pod \"a0a363bd-8ee8-4644-b685-e24ddfa39788\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") "
Oct 30 23:29:47.837512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837361    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-envoy\") pod \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") "
Oct 30 23:29:47.837512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837393    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-workload-socket\") pod \"a0a363bd-8ee8-4644-b685-e24ddfa39788\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") "
Oct 30 23:29:47.837512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837398    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-credential-socket" pod "a0a363bd-8ee8-4644-b685-e24ddfa39788" (UID: "a0a363bd-8ee8-4644-b685-e24ddfa39788"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:29:47.837512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837423    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-workload-socket\") pod \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") "
Oct 30 23:29:47.837512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837426    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a363bd-8ee8-4644-b685-e24ddfa39788-istiod-ca-cert" pod "a0a363bd-8ee8-4644-b685-e24ddfa39788" (UID: "a0a363bd-8ee8-4644-b685-e24ddfa39788"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:29:47.837512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837450    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-token\" (UniqueName: \"kubernetes.io/projected/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-token\") pod \"a0a363bd-8ee8-4644-b685-e24ddfa39788\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") "
Oct 30 23:29:47.838063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837575    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-workload-certs" pod "a0a363bd-8ee8-4644-b685-e24ddfa39788" (UID: "a0a363bd-8ee8-4644-b685-e24ddfa39788"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:29:47.838063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837591    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-token\" (UniqueName: \"kubernetes.io/projected/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-token\") pod \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") "
Oct 30 23:29:47.838063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837669    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-workload-certs\") pod \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") "
Oct 30 23:29:47.838063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837729    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-podinfo\") pod \"a0a363bd-8ee8-4644-b685-e24ddfa39788\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") "
Oct 30 23:29:47.838063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837732    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-workload-socket" pod "a0a363bd-8ee8-4644-b685-e24ddfa39788" (UID: "a0a363bd-8ee8-4644-b685-e24ddfa39788"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:29:47.838063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837777    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istiod-ca-cert\") pod \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\" (UID: \"84900961-7d7e-4bf0-9c40-6f3d0d8102a1\") "
Oct 30 23:29:47.838063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837806    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-data\") pod \"a0a363bd-8ee8-4644-b685-e24ddfa39788\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") "
Oct 30 23:29:47.838063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837854    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-envoy\") pod \"a0a363bd-8ee8-4644-b685-e24ddfa39788\" (UID: \"a0a363bd-8ee8-4644-b685-e24ddfa39788\") "
Oct 30 23:29:47.838471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837773    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-data" pod "84900961-7d7e-4bf0-9c40-6f3d0d8102a1" (UID: "84900961-7d7e-4bf0-9c40-6f3d0d8102a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:29:47.838471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.837952    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-credential-socket" pod "84900961-7d7e-4bf0-9c40-6f3d0d8102a1" (UID: "84900961-7d7e-4bf0-9c40-6f3d0d8102a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:29:47.838471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.838305    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istiod-ca-cert" pod "84900961-7d7e-4bf0-9c40-6f3d0d8102a1" (UID: "84900961-7d7e-4bf0-9c40-6f3d0d8102a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:29:47.838471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.838402    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-workload-certs" pod "84900961-7d7e-4bf0-9c40-6f3d0d8102a1" (UID: "84900961-7d7e-4bf0-9c40-6f3d0d8102a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:29:47.838471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.838456    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-data" pod "a0a363bd-8ee8-4644-b685-e24ddfa39788" (UID: "a0a363bd-8ee8-4644-b685-e24ddfa39788"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:29:47.838757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.838720    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:47.838757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.838741    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:47.838866 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.838756    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:47.838866 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.838770    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:47.838866 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.838784    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:47.838866 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.838801    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:47.839065 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.838872    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:47.839065 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.838885    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:47.839065 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.838898    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a363bd-8ee8-4644-b685-e24ddfa39788-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:47.839225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.839180    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-workload-socket" pod "84900961-7d7e-4bf0-9c40-6f3d0d8102a1" (UID: "84900961-7d7e-4bf0-9c40-6f3d0d8102a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:29:47.839937 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.839916    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-podinfo" pod "84900961-7d7e-4bf0-9c40-6f3d0d8102a1" (UID: "84900961-7d7e-4bf0-9c40-6f3d0d8102a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:29:47.840993 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.840959    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-podinfo" pod "a0a363bd-8ee8-4644-b685-e24ddfa39788" (UID: "a0a363bd-8ee8-4644-b685-e24ddfa39788"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:29:47.840993 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.840962    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-token" pod "84900961-7d7e-4bf0-9c40-6f3d0d8102a1" (UID: "84900961-7d7e-4bf0-9c40-6f3d0d8102a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:29:47.841321 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.841040    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a363bd-8ee8-4644-b685-e24ddfa39788-kube-api-access-ttmwf" pod "a0a363bd-8ee8-4644-b685-e24ddfa39788" (UID: "a0a363bd-8ee8-4644-b685-e24ddfa39788"). InnerVolumeSpecName "kube-api-access-ttmwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:29:47.841321 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.841118    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-kube-api-access-vz9vn" pod "84900961-7d7e-4bf0-9c40-6f3d0d8102a1" (UID: "84900961-7d7e-4bf0-9c40-6f3d0d8102a1"). InnerVolumeSpecName "kube-api-access-vz9vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:29:47.841321 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.841161    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-envoy" pod "a0a363bd-8ee8-4644-b685-e24ddfa39788" (UID: "a0a363bd-8ee8-4644-b685-e24ddfa39788"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:29:47.841716 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.841687    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-token" pod "a0a363bd-8ee8-4644-b685-e24ddfa39788" (UID: "a0a363bd-8ee8-4644-b685-e24ddfa39788"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:29:47.842553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.842532    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-envoy" pod "84900961-7d7e-4bf0-9c40-6f3d0d8102a1" (UID: "84900961-7d7e-4bf0-9c40-6f3d0d8102a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:29:47.940177 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.940143    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:47.940177 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.940173    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz9vn\" (UniqueName: \"kubernetes.io/projected/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-kube-api-access-vz9vn\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:47.940177 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.940184    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttmwf\" (UniqueName: \"kubernetes.io/projected/a0a363bd-8ee8-4644-b685-e24ddfa39788-kube-api-access-ttmwf\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:47.940480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.940192    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:47.940480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.940203    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:47.940480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.940218    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:47.940480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.940228    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84900961-7d7e-4bf0-9c40-6f3d0d8102a1-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:47.940480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.940236    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:47.940480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.940244    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a363bd-8ee8-4644-b685-e24ddfa39788-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:47.992725 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:47.992630    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm"]
Oct 30 23:29:48.008974 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.008939    5078 generic.go:356] "Generic (PLEG): container finished" podID="a0a363bd-8ee8-4644-b685-e24ddfa39788" containerID="930ef0b0be8f46366e681b7d050945d226e432828f3b401aab8d790b244dc0c8" exitCode=0
Oct 30 23:29:48.009450 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.008998    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"
Oct 30 23:29:48.009450 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.009035    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m" event={"ID":"a0a363bd-8ee8-4644-b685-e24ddfa39788","Type":"ContainerDied","Data":"930ef0b0be8f46366e681b7d050945d226e432828f3b401aab8d790b244dc0c8"}
Oct 30 23:29:48.009450 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.009106    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m" event={"ID":"a0a363bd-8ee8-4644-b685-e24ddfa39788","Type":"ContainerDied","Data":"4145bd7e800391e1eb862a2ac601da2917525550c3c51404c86e63d1dd0dc786"}
Oct 30 23:29:48.009450 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.009141    5078 scope.go:122] "RemoveContainer" containerID="930ef0b0be8f46366e681b7d050945d226e432828f3b401aab8d790b244dc0c8"
Oct 30 23:29:48.011045 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.011021    5078 generic.go:356] "Generic (PLEG): container finished" podID="84900961-7d7e-4bf0-9c40-6f3d0d8102a1" containerID="b89fe26163e55f5a2f54ec2c9934bdff2af9f6411da9f2916ae741e4bc33db41" exitCode=0
Oct 30 23:29:48.011571 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.011111    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"
Oct 30 23:29:48.011571 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.011462    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp" event={"ID":"84900961-7d7e-4bf0-9c40-6f3d0d8102a1","Type":"ContainerDied","Data":"b89fe26163e55f5a2f54ec2c9934bdff2af9f6411da9f2916ae741e4bc33db41"}
Oct 30 23:29:48.011571 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.011512    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp" event={"ID":"84900961-7d7e-4bf0-9c40-6f3d0d8102a1","Type":"ContainerDied","Data":"abc33f450293e953b8c6aabc0f3de8f0e59f7bd96fb8b361f80690b7b3e9cef1"}
Oct 30 23:29:48.048907 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.045674    5078 scope.go:122] "RemoveContainer" containerID="930ef0b0be8f46366e681b7d050945d226e432828f3b401aab8d790b244dc0c8"
Oct 30 23:29:48.048907 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:29:48.048117    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eebd647_3459_494d_b28c_cb27866e4551.slice/crio-525a0f7480edfd458e75d1f77d82e6895cabec4b32e51503ed383d0725ce7695 WatchSource:0}: Error finding container 525a0f7480edfd458e75d1f77d82e6895cabec4b32e51503ed383d0725ce7695: Status 404 returned error can't find the container with id 525a0f7480edfd458e75d1f77d82e6895cabec4b32e51503ed383d0725ce7695
Oct 30 23:29:48.048907 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:29:48.048277    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930ef0b0be8f46366e681b7d050945d226e432828f3b401aab8d790b244dc0c8\": container with ID starting with 930ef0b0be8f46366e681b7d050945d226e432828f3b401aab8d790b244dc0c8 not found: ID does not exist" containerID="930ef0b0be8f46366e681b7d050945d226e432828f3b401aab8d790b244dc0c8"
Oct 30 23:29:48.048907 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.048304    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930ef0b0be8f46366e681b7d050945d226e432828f3b401aab8d790b244dc0c8"} err="failed to get container status \"930ef0b0be8f46366e681b7d050945d226e432828f3b401aab8d790b244dc0c8\": rpc error: code = NotFound desc = could not find container \"930ef0b0be8f46366e681b7d050945d226e432828f3b401aab8d790b244dc0c8\": container with ID starting with 930ef0b0be8f46366e681b7d050945d226e432828f3b401aab8d790b244dc0c8 not found: ID does not exist"
Oct 30 23:29:48.048907 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.048323    5078 scope.go:122] "RemoveContainer" containerID="b89fe26163e55f5a2f54ec2c9934bdff2af9f6411da9f2916ae741e4bc33db41"
Oct 30 23:29:48.077245 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.077109    5078 scope.go:122] "RemoveContainer" containerID="b89fe26163e55f5a2f54ec2c9934bdff2af9f6411da9f2916ae741e4bc33db41"
Oct 30 23:29:48.078055 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:29:48.078028    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b89fe26163e55f5a2f54ec2c9934bdff2af9f6411da9f2916ae741e4bc33db41\": container with ID starting with b89fe26163e55f5a2f54ec2c9934bdff2af9f6411da9f2916ae741e4bc33db41 not found: ID does not exist" containerID="b89fe26163e55f5a2f54ec2c9934bdff2af9f6411da9f2916ae741e4bc33db41"
Oct 30 23:29:48.078143 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.078069    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b89fe26163e55f5a2f54ec2c9934bdff2af9f6411da9f2916ae741e4bc33db41"} err="failed to get container status \"b89fe26163e55f5a2f54ec2c9934bdff2af9f6411da9f2916ae741e4bc33db41\": rpc error: code = NotFound desc = could not find container \"b89fe26163e55f5a2f54ec2c9934bdff2af9f6411da9f2916ae741e4bc33db41\": container with ID starting with b89fe26163e55f5a2f54ec2c9934bdff2af9f6411da9f2916ae741e4bc33db41 not found: ID does not exist"
Oct 30 23:29:48.233629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.232086    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"]
Oct 30 23:29:48.275673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.274807    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-8nrmp"]
Oct 30 23:29:48.318393 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.318360    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84900961-7d7e-4bf0-9c40-6f3d0d8102a1" path="/var/lib/kubelet/pods/84900961-7d7e-4bf0-9c40-6f3d0d8102a1/volumes"
Oct 30 23:29:48.324857 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.324834    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp"
Oct 30 23:29:48.344516 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.344476    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"]
Oct 30 23:29:48.348697 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.348678    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-s9q2m"]
Oct 30 23:29:48.368705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.368683    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/528ef440-0f7c-4c42-b90f-55721ebf655b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/528ef440-0f7c-4c42-b90f-55721ebf655b-maas-api-xks-ca\") pod \"528ef440-0f7c-4c42-b90f-55721ebf655b\" (UID: \"528ef440-0f7c-4c42-b90f-55721ebf655b\") "
Oct 30 23:29:48.368952 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.368937    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/528ef440-0f7c-4c42-b90f-55721ebf655b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/528ef440-0f7c-4c42-b90f-55721ebf655b-maas-api-openshift-ca\") pod \"528ef440-0f7c-4c42-b90f-55721ebf655b\" (UID: \"528ef440-0f7c-4c42-b90f-55721ebf655b\") "
Oct 30 23:29:48.368997 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.368970    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528ef440-0f7c-4c42-b90f-55721ebf655b-kube-api-access-wq924\" (UniqueName: \"kubernetes.io/projected/528ef440-0f7c-4c42-b90f-55721ebf655b-kube-api-access-wq924\") pod \"528ef440-0f7c-4c42-b90f-55721ebf655b\" (UID: \"528ef440-0f7c-4c42-b90f-55721ebf655b\") "
Oct 30 23:29:48.369269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.369250    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528ef440-0f7c-4c42-b90f-55721ebf655b-maas-api-openshift-ca" pod "528ef440-0f7c-4c42-b90f-55721ebf655b" (UID: "528ef440-0f7c-4c42-b90f-55721ebf655b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:29:48.369482 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.369454    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/528ef440-0f7c-4c42-b90f-55721ebf655b-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:48.370820 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.370800    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/528ef440-0f7c-4c42-b90f-55721ebf655b-maas-api-xks-ca" pod "528ef440-0f7c-4c42-b90f-55721ebf655b" (UID: "528ef440-0f7c-4c42-b90f-55721ebf655b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:29:48.371119 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.371099    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528ef440-0f7c-4c42-b90f-55721ebf655b-kube-api-access-wq924" pod "528ef440-0f7c-4c42-b90f-55721ebf655b" (UID: "528ef440-0f7c-4c42-b90f-55721ebf655b"). InnerVolumeSpecName "kube-api-access-wq924". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:29:48.471001 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.470971    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wq924\" (UniqueName: \"kubernetes.io/projected/528ef440-0f7c-4c42-b90f-55721ebf655b-kube-api-access-wq924\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:48.471001 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:48.471002    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/528ef440-0f7c-4c42-b90f-55721ebf655b-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:29:49.028206 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:49.028172    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp"
Oct 30 23:29:49.028662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:49.028206    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp" event={"ID":"528ef440-0f7c-4c42-b90f-55721ebf655b","Type":"ContainerDied","Data":"af944d3f8f8bdd99bd5c5b86583068bcdf273b8bcfd473390850b9fc71705abb"}
Oct 30 23:29:49.028662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:49.028255    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af944d3f8f8bdd99bd5c5b86583068bcdf273b8bcfd473390850b9fc71705abb"
Oct 30 23:29:49.031041 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:49.031017    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm" event={"ID":"4eebd647-3459-494d-b28c-cb27866e4551","Type":"ContainerStarted","Data":"8a1a1874e1f5ff1464f2fa9c4bd48bfa13081fce4bc8ccecd22272f6fae63422"}
Oct 30 23:29:49.031166 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:49.031066    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm"
Oct 30 23:29:49.031166 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:49.031090    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm" event={"ID":"4eebd647-3459-494d-b28c-cb27866e4551","Type":"ContainerStarted","Data":"525a0f7480edfd458e75d1f77d82e6895cabec4b32e51503ed383d0725ce7695"}
Oct 30 23:29:49.184366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:49.184312    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm" podStartSLOduration=3.184298579 podStartE2EDuration="3.184298579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:29:49.152315831 +0000 UTC m=+1375.040222185" watchObservedRunningTime="2026-10-30 23:29:49.184298579 +0000 UTC m=+1375.072204934"
Oct 30 23:29:50.300101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:50.300068    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0a363bd-8ee8-4644-b685-e24ddfa39788" path="/var/lib/kubelet/pods/a0a363bd-8ee8-4644-b685-e24ddfa39788/volumes"
Oct 30 23:29:53.729765 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:53.729716    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:53.942625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:53.923691    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:53.942625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:53.924445    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:54.060751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:54.060699    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:54.175755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:54.175709    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:54.253633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:54.240419    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:54.565750 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:54.301959    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 30 23:29:55.102269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:54.566256    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:55.102269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:54.593854    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 30 23:29:55.102269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:54.607553    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded"
Oct 30 23:29:55.102269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:54.641564    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": context deadline exceeded"
Oct 30 23:29:55.102269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:54.756152    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:55.102269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:54.847995    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 23:29:55.102269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:54.848118    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 23:29:55.102269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:54.848165    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 23:29:55.102269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:54.848211    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:29:55.102269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:54.848237    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4" podUID="dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:55.102269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:54.848930    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:55.102269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:54.923701    5078 scope.go:122] "RemoveContainer" containerID="d82830e0410ded8c1f01ce1a0992c67a9025fe26ff1ca2bd4fe92a55d0438e2d"
Oct 30 23:29:55.102269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.029806    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:29:55.102269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.056669    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:55.102269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.056970    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:29:55.102269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.057867    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 23:29:55.102269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.058093    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:29:55.176432 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.176296    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 23:29:55.186806 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.186749    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:29:55.252880 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.252852    5078 scope.go:122] "RemoveContainer" containerID="5aed674adec47f6a26a98f31be24c4fd38b8e15b84ffa4455374785392d1d3a5"
Oct 30 23:29:55.297054 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.254711    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 23:29:55.297054 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.255634    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:29:55.351622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.330270    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": context deadline exceeded"
Oct 30 23:29:55.469624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.343839    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 30 23:29:55.469624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.468247    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:29:55.619622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.613325    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:55.619622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.614559    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:29:55.826694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.812283    5078 scope.go:122] "RemoveContainer" containerID="0a9c92f1997d85a6e13496d478cd82857323d0d28b93d6bad15cdebd9a68807f"
Oct 30 23:29:55.971684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.964873    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:55.971684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.965288    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:29:55.972856 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.972019    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:55.972856 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:55.972449    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:29:56.161865 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.155018    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:56.166622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.163242    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:56.171021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.167653    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:56.177225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.175380    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:56.201629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.181680    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:29:56.202003 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.201977    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:29:56.205621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.202191    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:29:56.225816 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.225782    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:29:56.348624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.184883    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:56.348624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.248337    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:56.348624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.183911    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Oct 30 23:29:56.348624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.184114    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 23:29:56.348624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.184614    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:56.493567 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.493369    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded"
Oct 30 23:29:56.520405 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.520358    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:56.672619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.657253    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:56.895703 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.878320    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 23:29:57.052252 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.956689    5078 scope.go:122] "RemoveContainer" containerID="f6f838a09165e60c2ab23d03801409cbd2cc5fa59e87c1b242c4bbf225f06a12"
Oct 30 23:29:57.052658 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:56.997789    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded"
Oct 30 23:29:57.066629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:57.018643    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 23:29:57.211128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:57.211031    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:57.212259 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:57.212221    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:57.212729 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:57.212704    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:57.221797 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:57.221767    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 23:29:57.414624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:57.377128    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:57.452623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:57.401837    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Oct 30 23:29:57.592364 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:57.401899    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 23:29:57.604924 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:57.401958    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 23:29:57.605268 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:29:57.484273    5078 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.161s"
Oct 30 23:29:57.605521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:57.591812    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:57.639708 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:57.639671    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl"]
Oct 30 23:29:57.794731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:57.794685    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:57.858847 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:57.858762    5078 scope.go:122] "RemoveContainer" containerID="272eecaee3983794b7a094a58beb98ef42bb4f6c8ceabe5400d5dabc6bd147b3"
Oct 30 23:29:57.963818 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:57.963774    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:58.106552 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.106511    5078 cpu_manager.go:414] "containerMap: removing container" podUID="528ef440-0f7c-4c42-b90f-55721ebf655b" containerName="revoke-keys"
Oct 30 23:29:58.106818 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.106797    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528ef440-0f7c-4c42-b90f-55721ebf655b" containerName="revoke-keys"
Oct 30 23:29:58.106988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.106974    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a0a363bd-8ee8-4644-b685-e24ddfa39788" containerName="istio-proxy"
Oct 30 23:29:58.107107 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.107094    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a363bd-8ee8-4644-b685-e24ddfa39788" containerName="istio-proxy"
Oct 30 23:29:58.107227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.107215    5078 cpu_manager.go:414] "containerMap: removing container" podUID="84900961-7d7e-4bf0-9c40-6f3d0d8102a1" containerName="istio-proxy"
Oct 30 23:29:58.107340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.107327    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84900961-7d7e-4bf0-9c40-6f3d0d8102a1" containerName="istio-proxy"
Oct 30 23:29:58.137965 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.137873    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="528ef440-0f7c-4c42-b90f-55721ebf655b" containerName="revoke-keys"
Oct 30 23:29:58.138196 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.138178    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a363bd-8ee8-4644-b685-e24ddfa39788" containerName="istio-proxy"
Oct 30 23:29:58.138341 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.138327    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="84900961-7d7e-4bf0-9c40-6f3d0d8102a1" containerName="istio-proxy"
Oct 30 23:29:58.173673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.140361    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:58.173673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.140650    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 23:29:58.173673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.140910    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded"
Oct 30 23:29:58.214512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.214462    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:58.216037 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.215104    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 23:29:58.225668 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.215144    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 23:29:58.230737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.230696    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:58.240141 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.240110    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr"]
Oct 30 23:29:58.461153 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.461059    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:58.464873 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.267479    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded"
Oct 30 23:29:58.529661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.298781    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl\" (UID: \"8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl"
Oct 30 23:29:58.529661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.266738    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl"
Oct 30 23:29:58.529661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.486037    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl"]
Oct 30 23:29:58.529661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.487113    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr"
Oct 30 23:29:58.529661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.528898    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8rfn\" (UniqueName: \"kubernetes.io/projected/8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf-kube-api-access-w8rfn\") pod \"payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl\" (UID: \"8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl"
Oct 30 23:29:58.705634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.677721    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:58.705634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.678272    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:58.731754 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.680744    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:58.731754 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.695726    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:58.731754 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.696657    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a68075cb-25c7-41ec-98ef-61ab9535c970-plugins-config-volume\") pod \"payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr\" (UID: \"a68075cb-25c7-41ec-98ef-61ab9535c970\") " pod="openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr"
Oct 30 23:29:58.731754 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.725004    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8rfn\" (UniqueName: \"kubernetes.io/projected/8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf-kube-api-access-w8rfn\") pod \"payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl\" (UID: \"8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl"
Oct 30 23:29:58.731754 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.725067    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl\" (UID: \"8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl"
Oct 30 23:29:58.731754 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.725346    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75cmh\" (UniqueName: \"kubernetes.io/projected/a68075cb-25c7-41ec-98ef-61ab9535c970-kube-api-access-75cmh\") pod \"payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr\" (UID: \"a68075cb-25c7-41ec-98ef-61ab9535c970\") " pod="openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr"
Oct 30 23:29:58.846092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.846057    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75cmh\" (UniqueName: \"kubernetes.io/projected/a68075cb-25c7-41ec-98ef-61ab9535c970-kube-api-access-75cmh\") pod \"payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr\" (UID: \"a68075cb-25c7-41ec-98ef-61ab9535c970\") " pod="openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr"
Oct 30 23:29:58.846436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.846413    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a68075cb-25c7-41ec-98ef-61ab9535c970-plugins-config-volume\") pod \"payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr\" (UID: \"a68075cb-25c7-41ec-98ef-61ab9535c970\") " pod="openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr"
Oct 30 23:29:58.847442 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.847411    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 30 23:29:58.855302 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.855231    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded"
Oct 30 23:29:58.855683 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.855661    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-01c440ca-dockercfg-ksb4x\""
Oct 30 23:29:58.858872 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.858852    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:29:58.859343 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.859169    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr"]
Oct 30 23:29:58.860180 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.860165    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:29:58.892428 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.870008    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-01c440ca\""
Oct 30 23:29:58.903365 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.903335    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8rfn\" (UniqueName: \"kubernetes.io/projected/8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf-kube-api-access-w8rfn\") pod \"payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl\" (UID: \"8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl"
Oct 30 23:29:58.912697 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.910775    5078 scope.go:122] "RemoveContainer" containerID="6dd2230b2c104860eb05092ffa07d36dd1ea033f2b50dcb95fa3cf86ff546737"
Oct 30 23:29:58.923040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.922892    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:29:58.923040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.922945    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:29:58.923040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.922978    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:29:58.943215 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.943176    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a68075cb-25c7-41ec-98ef-61ab9535c970-plugins-config-volume\") pod \"payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr\" (UID: \"a68075cb-25c7-41ec-98ef-61ab9535c970\") " pod="openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr"
Oct 30 23:29:58.945023 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.945000    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl\" (UID: \"8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl"
Oct 30 23:29:58.960677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:58.960654    5078 scope.go:122] "RemoveContainer" containerID="9812c77e0524c27ebf2db069c9065bb8126b7b0f65c832233fe966611ad4f4cf"
Oct 30 23:29:59.049938 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.044802    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75cmh\" (UniqueName: \"kubernetes.io/projected/a68075cb-25c7-41ec-98ef-61ab9535c970-kube-api-access-75cmh\") pod \"payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr\" (UID: \"a68075cb-25c7-41ec-98ef-61ab9535c970\") " pod="openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr"
Oct 30 23:29:59.068501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.060155    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:29:59.130042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.123278    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:59.133946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.133925    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:29:59.160843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.160812    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr"
Oct 30 23:29:59.172620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.168576    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:29:59.194951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.194918    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"]
Oct 30 23:29:59.212255 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.212224    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.216116 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.216083    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 23:29:59.232615 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.224163    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:29:59.233607 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.232945    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:29:59.233607 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.233390    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl"
Oct 30 23:29:59.296282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.296256    5078 scope.go:122] "RemoveContainer" containerID="4c66a6c566dfdce3fe349746466374faf2d95e8b98e1556e942158c55fe554ce"
Oct 30 23:29:59.298411 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.298300    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:29:59.302817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.302763    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-40c57b52-openshift-default-dockercfg-qsp98\""
Oct 30 23:29:59.310255 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.310090    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:29:59.310529 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.310508    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:29:59.315212 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.315184    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"]
Oct 30 23:29:59.317390 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.317374    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:29:59.341960 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.341931    5078 scope.go:122] "RemoveContainer" containerID="1ef07d85670047e01d4fb1b969141dcbffab8b2b7c71fad378f19e446ac165f7"
Oct 30 23:29:59.429620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.419950    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-credential-socket\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.429620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.420018    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-workload-socket\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.429620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.420057    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-podinfo\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.429620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.420164    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-data\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.429620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.420368    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-workload-certs\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.429620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.420403    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-token\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.429620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.420465    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c562aeb1-976f-4b17-b5b0-db336a33f67f-istiod-ca-cert\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.429620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.420496    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqx6s\" (UniqueName: \"kubernetes.io/projected/c562aeb1-976f-4b17-b5b0-db336a33f67f-kube-api-access-gqx6s\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.429620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.420531    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-envoy\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.533083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.528752    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-workload-certs\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.533083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.528803    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-token\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.533083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.528880    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c562aeb1-976f-4b17-b5b0-db336a33f67f-istiod-ca-cert\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.533083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.528920    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqx6s\" (UniqueName: \"kubernetes.io/projected/c562aeb1-976f-4b17-b5b0-db336a33f67f-kube-api-access-gqx6s\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.533083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.528957    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-envoy\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.533083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.529160    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-credential-socket\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.533083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.529201    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-workload-socket\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.533083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.529264    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-podinfo\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.533083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.529360    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-data\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.533083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.529892    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-data\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.533083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.530194    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-workload-certs\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.533083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.530975    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-credential-socket\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.533083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.531720    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c562aeb1-976f-4b17-b5b0-db336a33f67f-istiod-ca-cert\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.533083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.531801    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-workload-socket\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.536025 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.535936    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-envoy\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.539161 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.539128    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-podinfo\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.608783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.608698    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-token\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.628376 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.628347    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqx6s\" (UniqueName: \"kubernetes.io/projected/c562aeb1-976f-4b17-b5b0-db336a33f67f-kube-api-access-gqx6s\") pod \"e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") " pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:29:59.876632 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:29:59.846367    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:30:00.157907 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.157857    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm"
Oct 30 23:30:00.298395 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.267088    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr"]
Oct 30 23:30:00.431907 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.431875    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl"]
Oct 30 23:30:00.432040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.431939    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv"]
Oct 30 23:30:00.453498 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.453471    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh"]
Oct 30 23:30:00.455890 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.455866    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv"
Oct 30 23:30:00.462061 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.462036    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k"]
Oct 30 23:30:00.463992 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.463968    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh"
Oct 30 23:30:00.477999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.473403    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5"]
Oct 30 23:30:00.477999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.474826    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k"
Oct 30 23:30:00.509240 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.509210    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:30:00.509366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.509343    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:30:00.509471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.509453    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:30:00.518044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.517891    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5"
Oct 30 23:30:00.518440 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.518416    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf"]
Oct 30 23:30:00.529282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.529257    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh"]
Oct 30 23:30:00.529409 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.529309    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k"]
Oct 30 23:30:00.529409 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.529342    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5"]
Oct 30 23:30:00.529409 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.529369    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv"]
Oct 30 23:30:00.529409 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.529401    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf"]
Oct 30 23:30:00.529608 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.529493    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf"
Oct 30 23:30:00.541852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.541826    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x5vs\" (UniqueName: \"kubernetes.io/projected/bb6d7732-3d8a-470c-99c1-9bc2fabf83e9-kube-api-access-6x5vs\") pod \"maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh\" (UID: \"bb6d7732-3d8a-470c-99c1-9bc2fabf83e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh"
Oct 30 23:30:00.545663 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.545640    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5jxc\" (UniqueName: \"kubernetes.io/projected/b96df090-0d51-4d43-a5ac-ca40af4a29f2-kube-api-access-b5jxc\") pod \"maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv\" (UID: \"b96df090-0d51-4d43-a5ac-ca40af4a29f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv"
Oct 30 23:30:00.545865 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.545843    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4995\" (UniqueName: \"kubernetes.io/projected/b5c2142b-26d1-455e-a6cf-772a7204dbdb-kube-api-access-d4995\") pod \"maas-api-key-cleanup-29890050-t9n8k\" (UID: \"b5c2142b-26d1-455e-a6cf-772a7204dbdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k"
Oct 30 23:30:00.546048 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.546028    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6d7732-3d8a-470c-99c1-9bc2fabf83e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh\" (UID: \"bb6d7732-3d8a-470c-99c1-9bc2fabf83e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh"
Oct 30 23:30:00.546324 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.546304    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c2142b-26d1-455e-a6cf-772a7204dbdb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890050-t9n8k\" (UID: \"b5c2142b-26d1-455e-a6cf-772a7204dbdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k"
Oct 30 23:30:00.546880 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.546860    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96df090-0d51-4d43-a5ac-ca40af4a29f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv\" (UID: \"b96df090-0d51-4d43-a5ac-ca40af4a29f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv"
Oct 30 23:30:00.693672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.658991    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4855\" (UniqueName: \"kubernetes.io/projected/f3260b0f-1a11-4b46-bd73-71df2bb33461-kube-api-access-p4855\") pod \"maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf\" (UID: \"f3260b0f-1a11-4b46-bd73-71df2bb33461\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf"
Oct 30 23:30:00.744646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.704740    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x5vs\" (UniqueName: \"kubernetes.io/projected/bb6d7732-3d8a-470c-99c1-9bc2fabf83e9-kube-api-access-6x5vs\") pod \"maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh\" (UID: \"bb6d7732-3d8a-470c-99c1-9bc2fabf83e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh"
Oct 30 23:30:00.744646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.704829    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5jxc\" (UniqueName: \"kubernetes.io/projected/b96df090-0d51-4d43-a5ac-ca40af4a29f2-kube-api-access-b5jxc\") pod \"maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv\" (UID: \"b96df090-0d51-4d43-a5ac-ca40af4a29f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv"
Oct 30 23:30:00.744646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.704865    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e85026-f6e7-4cd8-9e1f-44e57104ca7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5\" (UID: \"36e85026-f6e7-4cd8-9e1f-44e57104ca7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5"
Oct 30 23:30:00.744646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.704896    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg97t\" (UniqueName: \"kubernetes.io/projected/36e85026-f6e7-4cd8-9e1f-44e57104ca7d-kube-api-access-fg97t\") pod \"maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5\" (UID: \"36e85026-f6e7-4cd8-9e1f-44e57104ca7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5"
Oct 30 23:30:00.744646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.704938    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4995\" (UniqueName: \"kubernetes.io/projected/b5c2142b-26d1-455e-a6cf-772a7204dbdb-kube-api-access-d4995\") pod \"maas-api-key-cleanup-29890050-t9n8k\" (UID: \"b5c2142b-26d1-455e-a6cf-772a7204dbdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k"
Oct 30 23:30:00.744646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.704986    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6d7732-3d8a-470c-99c1-9bc2fabf83e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh\" (UID: \"bb6d7732-3d8a-470c-99c1-9bc2fabf83e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh"
Oct 30 23:30:00.744646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.705116    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3260b0f-1a11-4b46-bd73-71df2bb33461-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf\" (UID: \"f3260b0f-1a11-4b46-bd73-71df2bb33461\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf"
Oct 30 23:30:00.744646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.705234    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c2142b-26d1-455e-a6cf-772a7204dbdb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890050-t9n8k\" (UID: \"b5c2142b-26d1-455e-a6cf-772a7204dbdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k"
Oct 30 23:30:00.744646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.705374    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96df090-0d51-4d43-a5ac-ca40af4a29f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv\" (UID: \"b96df090-0d51-4d43-a5ac-ca40af4a29f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv"
Oct 30 23:30:00.744646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.706284    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96df090-0d51-4d43-a5ac-ca40af4a29f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv\" (UID: \"b96df090-0d51-4d43-a5ac-ca40af4a29f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv"
Oct 30 23:30:00.744646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.707532    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6d7732-3d8a-470c-99c1-9bc2fabf83e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh\" (UID: \"bb6d7732-3d8a-470c-99c1-9bc2fabf83e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh"
Oct 30 23:30:00.744646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.708301    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c2142b-26d1-455e-a6cf-772a7204dbdb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890050-t9n8k\" (UID: \"b5c2142b-26d1-455e-a6cf-772a7204dbdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k"
Oct 30 23:30:00.852390 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.852276    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4855\" (UniqueName: \"kubernetes.io/projected/f3260b0f-1a11-4b46-bd73-71df2bb33461-kube-api-access-p4855\") pod \"maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf\" (UID: \"f3260b0f-1a11-4b46-bd73-71df2bb33461\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf"
Oct 30 23:30:00.852904 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.852879    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e85026-f6e7-4cd8-9e1f-44e57104ca7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5\" (UID: \"36e85026-f6e7-4cd8-9e1f-44e57104ca7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5"
Oct 30 23:30:00.853057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.853039    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg97t\" (UniqueName: \"kubernetes.io/projected/36e85026-f6e7-4cd8-9e1f-44e57104ca7d-kube-api-access-fg97t\") pod \"maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5\" (UID: \"36e85026-f6e7-4cd8-9e1f-44e57104ca7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5"
Oct 30 23:30:00.853220 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.853206    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3260b0f-1a11-4b46-bd73-71df2bb33461-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf\" (UID: \"f3260b0f-1a11-4b46-bd73-71df2bb33461\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf"
Oct 30 23:30:00.855947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.855410    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3260b0f-1a11-4b46-bd73-71df2bb33461-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf\" (UID: \"f3260b0f-1a11-4b46-bd73-71df2bb33461\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf"
Oct 30 23:30:00.889704 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.889654    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e85026-f6e7-4cd8-9e1f-44e57104ca7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5\" (UID: \"36e85026-f6e7-4cd8-9e1f-44e57104ca7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5"
Oct 30 23:30:00.896833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.896805    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6"]
Oct 30 23:30:00.900624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.900580    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4995\" (UniqueName: \"kubernetes.io/projected/b5c2142b-26d1-455e-a6cf-772a7204dbdb-kube-api-access-d4995\") pod \"maas-api-key-cleanup-29890050-t9n8k\" (UID: \"b5c2142b-26d1-455e-a6cf-772a7204dbdb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k"
Oct 30 23:30:00.901236 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.901209    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5jxc\" (UniqueName: \"kubernetes.io/projected/b96df090-0d51-4d43-a5ac-ca40af4a29f2-kube-api-access-b5jxc\") pod \"maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv\" (UID: \"b96df090-0d51-4d43-a5ac-ca40af4a29f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv"
Oct 30 23:30:00.903174 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.903119    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x5vs\" (UniqueName: \"kubernetes.io/projected/bb6d7732-3d8a-470c-99c1-9bc2fabf83e9-kube-api-access-6x5vs\") pod \"maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh\" (UID: \"bb6d7732-3d8a-470c-99c1-9bc2fabf83e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh"
Oct 30 23:30:00.911898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.911876    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4855\" (UniqueName: \"kubernetes.io/projected/f3260b0f-1a11-4b46-bd73-71df2bb33461-kube-api-access-p4855\") pod \"maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf\" (UID: \"f3260b0f-1a11-4b46-bd73-71df2bb33461\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf"
Oct 30 23:30:00.912113 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.912006    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg97t\" (UniqueName: \"kubernetes.io/projected/36e85026-f6e7-4cd8-9e1f-44e57104ca7d-kube-api-access-fg97t\") pod \"maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5\" (UID: \"36e85026-f6e7-4cd8-9e1f-44e57104ca7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5"
Oct 30 23:30:00.914089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.914059    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6"]
Oct 30 23:30:00.914192 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.914169    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6"
Oct 30 23:30:00.959556 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.959521    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvcp\" (UniqueName: \"kubernetes.io/projected/ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608-kube-api-access-wcvcp\") pod \"payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6\" (UID: \"ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608\") " pod="openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6"
Oct 30 23:30:00.959837 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.959815    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608-plugins-config-volume\") pod \"payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6\" (UID: \"ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608\") " pod="openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6"
Oct 30 23:30:00.963406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:30:00.963382    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc562aeb1_976f_4b17_b5b0_db336a33f67f.slice/crio-98c98c4104c31e856e9a184307858b02611256cefe0fd84415ace8bf1d4f0c44 WatchSource:0}: Error finding container 98c98c4104c31e856e9a184307858b02611256cefe0fd84415ace8bf1d4f0c44: Status 404 returned error can't find the container with id 98c98c4104c31e856e9a184307858b02611256cefe0fd84415ace8bf1d4f0c44
Oct 30 23:30:00.964982 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.964967    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"]
Oct 30 23:30:00.966679 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.966657    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2b6c82a0-dockercfg-tqrw7\""
Oct 30 23:30:00.967259 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.967221    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:30:00.967355 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.967304    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:30:00.967355 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.967346    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:30:00.967633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:00.967615    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2b6c82a0\""
Oct 30 23:30:01.052441 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.052417    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k"
Oct 30 23:30:01.061783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.061758    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608-plugins-config-volume\") pod \"payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6\" (UID: \"ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608\") " pod="openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6"
Oct 30 23:30:01.061945 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.061929    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvcp\" (UniqueName: \"kubernetes.io/projected/ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608-kube-api-access-wcvcp\") pod \"payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6\" (UID: \"ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608\") " pod="openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6"
Oct 30 23:30:01.062742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.062714    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608-plugins-config-volume\") pod \"payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6\" (UID: \"ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608\") " pod="openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6"
Oct 30 23:30:01.093490 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.093451    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv"
Oct 30 23:30:01.119964 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.119899    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvcp\" (UniqueName: \"kubernetes.io/projected/ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608-kube-api-access-wcvcp\") pod \"payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6\" (UID: \"ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608\") " pod="openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6"
Oct 30 23:30:01.130993 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.130963    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh"
Oct 30 23:30:01.155351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.155323    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5"
Oct 30 23:30:01.201270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.201235    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf"
Oct 30 23:30:01.260246 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.259989    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6"
Oct 30 23:30:01.424402 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.423386    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr" event={"ID":"a68075cb-25c7-41ec-98ef-61ab9535c970","Type":"ContainerStarted","Data":"d3c448ffcff1aefcefe99679ae72dc1150e417b72973fb4742744e33c53b5f9c"}
Oct 30 23:30:01.424402 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.423447    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr" event={"ID":"a68075cb-25c7-41ec-98ef-61ab9535c970","Type":"ContainerStarted","Data":"d4cc25ce259f2ec5b4b048c993a666291c4f6b745db6b08ed34773560e7bb9df"}
Oct 30 23:30:01.424402 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.424336    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr"
Oct 30 23:30:01.427630 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.427477    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl" event={"ID":"8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf","Type":"ContainerStarted","Data":"218d63fa936ad92c96ba922783092f2e70ac43108c35cb3d25774ca30259f3c3"}
Oct 30 23:30:01.427630 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.427532    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl" event={"ID":"8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf","Type":"ContainerStarted","Data":"11fe09de98db643edddba24a6f90c06218bc23177648501a6db1939b33a84194"}
Oct 30 23:30:01.427630 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.427575    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl"
Oct 30 23:30:01.430041 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.429999    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9" event={"ID":"c562aeb1-976f-4b17-b5b0-db336a33f67f","Type":"ContainerStarted","Data":"4109cab6a90085e24af88132916d95de0d4b1dbac88999a104ab211c3ef12601"}
Oct 30 23:30:01.430176 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.430043    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9" event={"ID":"c562aeb1-976f-4b17-b5b0-db336a33f67f","Type":"ContainerStarted","Data":"98c98c4104c31e856e9a184307858b02611256cefe0fd84415ace8bf1d4f0c44"}
Oct 30 23:30:01.490554 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.490524    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k"]
Oct 30 23:30:01.491803 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:30:01.491772    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c2142b_26d1_455e_a6cf_772a7204dbdb.slice/crio-63bbc8007dc33454ea59a03758abd5f9265b00296cfa2e5ae60277ea80574609 WatchSource:0}: Error finding container 63bbc8007dc33454ea59a03758abd5f9265b00296cfa2e5ae60277ea80574609: Status 404 returned error can't find the container with id 63bbc8007dc33454ea59a03758abd5f9265b00296cfa2e5ae60277ea80574609
Oct 30 23:30:01.612804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.612774    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh"]
Oct 30 23:30:01.660290 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:30:01.660254    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb6d7732_3d8a_470c_99c1_9bc2fabf83e9.slice/crio-865d7d53d6a411d6c58bcc07d06be407dca9c7b473a57332f4c9a7556c67f13a WatchSource:0}: Error finding container 865d7d53d6a411d6c58bcc07d06be407dca9c7b473a57332f4c9a7556c67f13a: Status 404 returned error can't find the container with id 865d7d53d6a411d6c58bcc07d06be407dca9c7b473a57332f4c9a7556c67f13a
Oct 30 23:30:01.696945 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:30:01.696903    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3260b0f_1a11_4b46_bd73_71df2bb33461.slice/crio-5d1a15defa8b94495adef4527d3e30f773bd7e2eb66e5b06ab2539e00163a157 WatchSource:0}: Error finding container 5d1a15defa8b94495adef4527d3e30f773bd7e2eb66e5b06ab2539e00163a157: Status 404 returned error can't find the container with id 5d1a15defa8b94495adef4527d3e30f773bd7e2eb66e5b06ab2539e00163a157
Oct 30 23:30:01.699702 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.699676    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf"]
Oct 30 23:30:01.709523 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.709483    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr" podStartSLOduration=6.709469622 podStartE2EDuration="6.709469622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:01.708153109 +0000 UTC m=+1387.596059467" watchObservedRunningTime="2026-10-30 23:30:01.709469622 +0000 UTC m=+1387.597375975"
Oct 30 23:30:01.749153 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:30:01.749121    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96df090_0d51_4d43_a5ac_ca40af4a29f2.slice/crio-9680939e5a6a948a67fbf9b078672ee136b306a374521db7afe4ec95c28b82b9 WatchSource:0}: Error finding container 9680939e5a6a948a67fbf9b078672ee136b306a374521db7afe4ec95c28b82b9: Status 404 returned error can't find the container with id 9680939e5a6a948a67fbf9b078672ee136b306a374521db7afe4ec95c28b82b9
Oct 30 23:30:01.749779 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.749753    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv"]
Oct 30 23:30:01.758079 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:30:01.757770    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3b0cd5_173f_4bc5_a7bf_c4fecf81e608.slice/crio-ff648774cbe651af2d1656009a396197da45ac76fbfe2cffd5852a5f263142d0 WatchSource:0}: Error finding container ff648774cbe651af2d1656009a396197da45ac76fbfe2cffd5852a5f263142d0: Status 404 returned error can't find the container with id ff648774cbe651af2d1656009a396197da45ac76fbfe2cffd5852a5f263142d0
Oct 30 23:30:01.758911 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.758881    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6"]
Oct 30 23:30:01.866031 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.866001    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:30:01.871091 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.871034    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5"]
Oct 30 23:30:01.960997 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:01.960675    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9" podStartSLOduration=2.960576144 podStartE2EDuration="2.960576144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:01.953256621 +0000 UTC m=+1387.841162974" watchObservedRunningTime="2026-10-30 23:30:01.960576144 +0000 UTC m=+1387.848482498"
Oct 30 23:30:02.201388 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:02.201309    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl" podStartSLOduration=7.201284099 podStartE2EDuration="7.201284099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:02.075617228 +0000 UTC m=+1387.963523575" watchObservedRunningTime="2026-10-30 23:30:02.201284099 +0000 UTC m=+1388.089190492"
Oct 30 23:30:02.441121 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:02.441087    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6" event={"ID":"ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608","Type":"ContainerStarted","Data":"68bf0181b17594db932b42af3e66a7f476a227a9a5ee07d8fa6f37ff125bd9b1"}
Oct 30 23:30:02.441548 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:02.441146    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6"
Oct 30 23:30:02.441548 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:02.441168    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6" event={"ID":"ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608","Type":"ContainerStarted","Data":"ff648774cbe651af2d1656009a396197da45ac76fbfe2cffd5852a5f263142d0"}
Oct 30 23:30:02.442894 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:02.442869    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh" event={"ID":"bb6d7732-3d8a-470c-99c1-9bc2fabf83e9","Type":"ContainerStarted","Data":"4bb033a23db5ce709799b012ab894be12d1e79256c1b964199eda8a0a57f1152"}
Oct 30 23:30:02.443039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:02.442917    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh" event={"ID":"bb6d7732-3d8a-470c-99c1-9bc2fabf83e9","Type":"ContainerStarted","Data":"865d7d53d6a411d6c58bcc07d06be407dca9c7b473a57332f4c9a7556c67f13a"}
Oct 30 23:30:02.444543 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:02.444521    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf" event={"ID":"f3260b0f-1a11-4b46-bd73-71df2bb33461","Type":"ContainerStarted","Data":"2c53609d92407824ac46e315b5ded4a0d5ac3f2980f34408fb242befa03fe6ff"}
Oct 30 23:30:02.444695 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:02.444569    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf" event={"ID":"f3260b0f-1a11-4b46-bd73-71df2bb33461","Type":"ContainerStarted","Data":"5d1a15defa8b94495adef4527d3e30f773bd7e2eb66e5b06ab2539e00163a157"}
Oct 30 23:30:02.446235 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:02.446207    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5" event={"ID":"36e85026-f6e7-4cd8-9e1f-44e57104ca7d","Type":"ContainerStarted","Data":"ea432067e37617357dd4cc6990a01d08ef7b3cf8feea22e7906c19920848341e"}
Oct 30 23:30:02.446345 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:02.446252    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5" event={"ID":"36e85026-f6e7-4cd8-9e1f-44e57104ca7d","Type":"ContainerStarted","Data":"3119de5fa0ff209269f2c866b3147cc0232cefcd51003b3009baf03ec5f71c04"}
Oct 30 23:30:02.447474 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:02.447448    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv" event={"ID":"b96df090-0d51-4d43-a5ac-ca40af4a29f2","Type":"ContainerStarted","Data":"9680939e5a6a948a67fbf9b078672ee136b306a374521db7afe4ec95c28b82b9"}
Oct 30 23:30:02.449054 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:02.449029    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k" event={"ID":"b5c2142b-26d1-455e-a6cf-772a7204dbdb","Type":"ContainerStarted","Data":"86f1b8cf27246d28c9822b16f196861748ec6112ae64c11324eca1abeae65140"}
Oct 30 23:30:02.449129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:02.449073    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k" event={"ID":"b5c2142b-26d1-455e-a6cf-772a7204dbdb","Type":"ContainerStarted","Data":"63bbc8007dc33454ea59a03758abd5f9265b00296cfa2e5ae60277ea80574609"}
Oct 30 23:30:02.774891 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:02.774830    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5" podStartSLOduration=2.774809948 podStartE2EDuration="2.774809948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:02.770799021 +0000 UTC m=+1388.658705384" watchObservedRunningTime="2026-10-30 23:30:02.774809948 +0000 UTC m=+1388.662716303"
Oct 30 23:30:02.871040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:02.871012    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:30:02.885167 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:02.885108    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k" podStartSLOduration=2.885087863 podStartE2EDuration="2.885087863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:02.884120011 +0000 UTC m=+1388.772026403" watchObservedRunningTime="2026-10-30 23:30:02.885087863 +0000 UTC m=+1388.772994216"
Oct 30 23:30:03.005074 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.004972    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6" podStartSLOduration=3.004956722 podStartE2EDuration="3.004956722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:03.000269776 +0000 UTC m=+1388.888176162" watchObservedRunningTime="2026-10-30 23:30:03.004956722 +0000 UTC m=+1388.892863074"
Oct 30 23:30:03.263975 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.263871    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh" podStartSLOduration=3.263855223 podStartE2EDuration="3.263855223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:03.261257926 +0000 UTC m=+1389.149164279" watchObservedRunningTime="2026-10-30 23:30:03.263855223 +0000 UTC m=+1389.151761604"
Oct 30 23:30:03.266034 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.265994    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf" podStartSLOduration=3.265979579 podStartE2EDuration="3.265979579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:03.143702782 +0000 UTC m=+1389.031609135" watchObservedRunningTime="2026-10-30 23:30:03.265979579 +0000 UTC m=+1389.153885980"
Oct 30 23:30:03.458664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.458634    5078 generic.go:356] "Generic (PLEG): container finished" podID="bb6d7732-3d8a-470c-99c1-9bc2fabf83e9" containerID="4bb033a23db5ce709799b012ab894be12d1e79256c1b964199eda8a0a57f1152" exitCode=60
Oct 30 23:30:03.459111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.458736    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh" event={"ID":"bb6d7732-3d8a-470c-99c1-9bc2fabf83e9","Type":"ContainerDied","Data":"4bb033a23db5ce709799b012ab894be12d1e79256c1b964199eda8a0a57f1152"}
Oct 30 23:30:03.459111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.459098    5078 scope.go:122] "RemoveContainer" containerID="4bb033a23db5ce709799b012ab894be12d1e79256c1b964199eda8a0a57f1152"
Oct 30 23:30:03.462570 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.462545    5078 generic.go:356] "Generic (PLEG): container finished" podID="f3260b0f-1a11-4b46-bd73-71df2bb33461" containerID="2c53609d92407824ac46e315b5ded4a0d5ac3f2980f34408fb242befa03fe6ff" exitCode=60
Oct 30 23:30:03.462730 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.462633    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf" event={"ID":"f3260b0f-1a11-4b46-bd73-71df2bb33461","Type":"ContainerDied","Data":"2c53609d92407824ac46e315b5ded4a0d5ac3f2980f34408fb242befa03fe6ff"}
Oct 30 23:30:03.463700 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.463135    5078 scope.go:122] "RemoveContainer" containerID="2c53609d92407824ac46e315b5ded4a0d5ac3f2980f34408fb242befa03fe6ff"
Oct 30 23:30:03.465480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.465459    5078 generic.go:356] "Generic (PLEG): container finished" podID="36e85026-f6e7-4cd8-9e1f-44e57104ca7d" containerID="ea432067e37617357dd4cc6990a01d08ef7b3cf8feea22e7906c19920848341e" exitCode=60
Oct 30 23:30:03.465567 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.465508    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5" event={"ID":"36e85026-f6e7-4cd8-9e1f-44e57104ca7d","Type":"ContainerDied","Data":"ea432067e37617357dd4cc6990a01d08ef7b3cf8feea22e7906c19920848341e"}
Oct 30 23:30:03.465805 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.465789    5078 scope.go:122] "RemoveContainer" containerID="ea432067e37617357dd4cc6990a01d08ef7b3cf8feea22e7906c19920848341e"
Oct 30 23:30:03.467365 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.467347    5078 generic.go:356] "Generic (PLEG): container finished" podID="b96df090-0d51-4d43-a5ac-ca40af4a29f2" containerID="5e846a83bed73217ec37591eb77ec6c4e5106e10713c44214856586fc69e696a" exitCode=60
Oct 30 23:30:03.467477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.467436    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv" event={"ID":"b96df090-0d51-4d43-a5ac-ca40af4a29f2","Type":"ContainerDied","Data":"5e846a83bed73217ec37591eb77ec6c4e5106e10713c44214856586fc69e696a"}
Oct 30 23:30:03.467678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.467664    5078 scope.go:122] "RemoveContainer" containerID="5e846a83bed73217ec37591eb77ec6c4e5106e10713c44214856586fc69e696a"
Oct 30 23:30:03.469146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.469124    5078 generic.go:356] "Generic (PLEG): container finished" podID="b5c2142b-26d1-455e-a6cf-772a7204dbdb" containerID="86f1b8cf27246d28c9822b16f196861748ec6112ae64c11324eca1abeae65140" exitCode=60
Oct 30 23:30:03.469221 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.469199    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k" event={"ID":"b5c2142b-26d1-455e-a6cf-772a7204dbdb","Type":"ContainerDied","Data":"86f1b8cf27246d28c9822b16f196861748ec6112ae64c11324eca1abeae65140"}
Oct 30 23:30:03.469390 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.469375    5078 scope.go:122] "RemoveContainer" containerID="86f1b8cf27246d28c9822b16f196861748ec6112ae64c11324eca1abeae65140"
Oct 30 23:30:03.471232 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.470796    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:30:03.472946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:03.472889    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:30:04.480066 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:04.480029    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k" event={"ID":"b5c2142b-26d1-455e-a6cf-772a7204dbdb","Type":"ContainerStarted","Data":"53e68d97178639db300c78712ff82e006cce581027741b58ea7e6ddba9fb2668"}
Oct 30 23:30:04.482025 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:04.481999    5078 generic.go:356] "Generic (PLEG): container finished" podID="bb6d7732-3d8a-470c-99c1-9bc2fabf83e9" containerID="d4acd5e9a49d4ac85ca33638373413b69ea0c3e4b62db195f259f2adc5d5b0fb" exitCode=60
Oct 30 23:30:04.482130 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:04.482078    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh" event={"ID":"bb6d7732-3d8a-470c-99c1-9bc2fabf83e9","Type":"ContainerDied","Data":"d4acd5e9a49d4ac85ca33638373413b69ea0c3e4b62db195f259f2adc5d5b0fb"}
Oct 30 23:30:04.482170 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:04.482129    5078 scope.go:122] "RemoveContainer" containerID="4bb033a23db5ce709799b012ab894be12d1e79256c1b964199eda8a0a57f1152"
Oct 30 23:30:04.482374 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:04.482359    5078 scope.go:122] "RemoveContainer" containerID="d4acd5e9a49d4ac85ca33638373413b69ea0c3e4b62db195f259f2adc5d5b0fb"
Oct 30 23:30:04.482768 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:30:04.482745    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh_odh-ai-gateway-infra(bb6d7732-3d8a-470c-99c1-9bc2fabf83e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh" podUID="bb6d7732-3d8a-470c-99c1-9bc2fabf83e9"
Oct 30 23:30:04.484328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:04.484296    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf" event={"ID":"f3260b0f-1a11-4b46-bd73-71df2bb33461","Type":"ContainerStarted","Data":"f46d2364fa8c21fcd94f6a5887734ebc7024f88b98a6e256d99069a815da42d7"}
Oct 30 23:30:04.486079 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:04.486055    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5" event={"ID":"36e85026-f6e7-4cd8-9e1f-44e57104ca7d","Type":"ContainerStarted","Data":"b7825ff61bc1879ecfd710f957dccc53bf69bf154594d71fdc60e30795d40121"}
Oct 30 23:30:04.488203 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:04.488182    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv" event={"ID":"b96df090-0d51-4d43-a5ac-ca40af4a29f2","Type":"ContainerStarted","Data":"7447e458ccf510e2d2eabb9015fc0222ba463db3ac70634239352d6e6c13a47f"}
Oct 30 23:30:04.949570 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:04.949508    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv" podStartSLOduration=4.949494073 podStartE2EDuration="4.949494073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:04.945552941 +0000 UTC m=+1390.833459293" watchObservedRunningTime="2026-10-30 23:30:04.949494073 +0000 UTC m=+1390.837400420"
Oct 30 23:30:05.512812 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.512781    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n"]
Oct 30 23:30:05.540819 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.540790    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n"]
Oct 30 23:30:05.541114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.541100    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n"
Oct 30 23:30:05.672542 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.623377    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-5f9ec6be\""
Oct 30 23:30:05.672542 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.624137    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-5f9ec6be-dockercfg-w2x9g\""
Oct 30 23:30:05.672542 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.638372    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q279\" (UniqueName: \"kubernetes.io/projected/0471065d-e2f3-4d32-83df-90c783556e1d-kube-api-access-8q279\") pod \"payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n\" (UID: \"0471065d-e2f3-4d32-83df-90c783556e1d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n"
Oct 30 23:30:05.672542 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.638663    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0471065d-e2f3-4d32-83df-90c783556e1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n\" (UID: \"0471065d-e2f3-4d32-83df-90c783556e1d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n"
Oct 30 23:30:05.723018 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.722970    5078 generic.go:356] "Generic (PLEG): container finished" podID="b5c2142b-26d1-455e-a6cf-772a7204dbdb" containerID="53e68d97178639db300c78712ff82e006cce581027741b58ea7e6ddba9fb2668" exitCode=60
Oct 30 23:30:05.726220 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.726200    5078 scope.go:122] "RemoveContainer" containerID="53e68d97178639db300c78712ff82e006cce581027741b58ea7e6ddba9fb2668"
Oct 30 23:30:05.726965 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:30:05.726938    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890050-t9n8k_odh-ai-gateway-infra(b5c2142b-26d1-455e-a6cf-772a7204dbdb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k" podUID="b5c2142b-26d1-455e-a6cf-772a7204dbdb"
Oct 30 23:30:05.732049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.732027    5078 scope.go:122] "RemoveContainer" containerID="d4acd5e9a49d4ac85ca33638373413b69ea0c3e4b62db195f259f2adc5d5b0fb"
Oct 30 23:30:05.732718 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:30:05.732695    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh_odh-ai-gateway-infra(bb6d7732-3d8a-470c-99c1-9bc2fabf83e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh" podUID="bb6d7732-3d8a-470c-99c1-9bc2fabf83e9"
Oct 30 23:30:05.743220 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.733929    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k" event={"ID":"b5c2142b-26d1-455e-a6cf-772a7204dbdb","Type":"ContainerDied","Data":"53e68d97178639db300c78712ff82e006cce581027741b58ea7e6ddba9fb2668"}
Oct 30 23:30:05.743472 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.743451    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf" event={"ID":"f3260b0f-1a11-4b46-bd73-71df2bb33461","Type":"ContainerDied","Data":"f46d2364fa8c21fcd94f6a5887734ebc7024f88b98a6e256d99069a815da42d7"}
Oct 30 23:30:05.743639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.737314    5078 scope.go:122] "RemoveContainer" containerID="f46d2364fa8c21fcd94f6a5887734ebc7024f88b98a6e256d99069a815da42d7"
Oct 30 23:30:05.743815 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.743789    5078 scope.go:122] "RemoveContainer" containerID="86f1b8cf27246d28c9822b16f196861748ec6112ae64c11324eca1abeae65140"
Oct 30 23:30:05.743978 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.742185    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0471065d-e2f3-4d32-83df-90c783556e1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n\" (UID: \"0471065d-e2f3-4d32-83df-90c783556e1d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n"
Oct 30 23:30:05.744291 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.744273    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q279\" (UniqueName: \"kubernetes.io/projected/0471065d-e2f3-4d32-83df-90c783556e1d-kube-api-access-8q279\") pod \"payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n\" (UID: \"0471065d-e2f3-4d32-83df-90c783556e1d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n"
Oct 30 23:30:05.744974 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.743154    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0471065d-e2f3-4d32-83df-90c783556e1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n\" (UID: \"0471065d-e2f3-4d32-83df-90c783556e1d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n"
Oct 30 23:30:05.745130 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.736853    5078 generic.go:356] "Generic (PLEG): container finished" podID="f3260b0f-1a11-4b46-bd73-71df2bb33461" containerID="f46d2364fa8c21fcd94f6a5887734ebc7024f88b98a6e256d99069a815da42d7" exitCode=60
Oct 30 23:30:05.748031 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:30:05.748006    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf_odh-ai-gateway-infra(f3260b0f-1a11-4b46-bd73-71df2bb33461)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf" podUID="f3260b0f-1a11-4b46-bd73-71df2bb33461"
Oct 30 23:30:05.753474 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.753443    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn"]
Oct 30 23:30:05.760020 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.759995    5078 generic.go:356] "Generic (PLEG): container finished" podID="36e85026-f6e7-4cd8-9e1f-44e57104ca7d" containerID="b7825ff61bc1879ecfd710f957dccc53bf69bf154594d71fdc60e30795d40121" exitCode=60
Oct 30 23:30:05.760769 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.760753    5078 scope.go:122] "RemoveContainer" containerID="b7825ff61bc1879ecfd710f957dccc53bf69bf154594d71fdc60e30795d40121"
Oct 30 23:30:05.761671 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:30:05.761644    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5_odh-ai-gateway-infra(36e85026-f6e7-4cd8-9e1f-44e57104ca7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5" podUID="36e85026-f6e7-4cd8-9e1f-44e57104ca7d"
Oct 30 23:30:05.776554 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.776349    5078 generic.go:356] "Generic (PLEG): container finished" podID="b96df090-0d51-4d43-a5ac-ca40af4a29f2" containerID="7447e458ccf510e2d2eabb9015fc0222ba463db3ac70634239352d6e6c13a47f" exitCode=60
Oct 30 23:30:05.782978 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.777970    5078 scope.go:122] "RemoveContainer" containerID="7447e458ccf510e2d2eabb9015fc0222ba463db3ac70634239352d6e6c13a47f"
Oct 30 23:30:05.782978 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:30:05.778298    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv_odh-ai-gateway-infra(b96df090-0d51-4d43-a5ac-ca40af4a29f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv" podUID="b96df090-0d51-4d43-a5ac-ca40af4a29f2"
Oct 30 23:30:05.792849 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.792821    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn"]
Oct 30 23:30:05.792994 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.792869    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5" event={"ID":"36e85026-f6e7-4cd8-9e1f-44e57104ca7d","Type":"ContainerDied","Data":"b7825ff61bc1879ecfd710f957dccc53bf69bf154594d71fdc60e30795d40121"}
Oct 30 23:30:05.792994 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.792906    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv" event={"ID":"b96df090-0d51-4d43-a5ac-ca40af4a29f2","Type":"ContainerDied","Data":"7447e458ccf510e2d2eabb9015fc0222ba463db3ac70634239352d6e6c13a47f"}
Oct 30 23:30:05.793141 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.793011    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn"
Oct 30 23:30:05.825622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.810948    5078 scope.go:122] "RemoveContainer" containerID="2c53609d92407824ac46e315b5ded4a0d5ac3f2980f34408fb242befa03fe6ff"
Oct 30 23:30:05.861897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.861862    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea6ae1c-d046-499e-89c6-22a626aa98c4-plugins-config-volume\") pod \"payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn\" (UID: \"4ea6ae1c-d046-499e-89c6-22a626aa98c4\") " pod="openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn"
Oct 30 23:30:05.877032 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.876996    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsm5g\" (UniqueName: \"kubernetes.io/projected/4ea6ae1c-d046-499e-89c6-22a626aa98c4-kube-api-access-xsm5g\") pod \"payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn\" (UID: \"4ea6ae1c-d046-499e-89c6-22a626aa98c4\") " pod="openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn"
Oct 30 23:30:05.907354 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.907323    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q279\" (UniqueName: \"kubernetes.io/projected/0471065d-e2f3-4d32-83df-90c783556e1d-kube-api-access-8q279\") pod \"payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n\" (UID: \"0471065d-e2f3-4d32-83df-90c783556e1d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n"
Oct 30 23:30:05.921739 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:05.921344    5078 scope.go:122] "RemoveContainer" containerID="ea432067e37617357dd4cc6990a01d08ef7b3cf8feea22e7906c19920848341e"
Oct 30 23:30:06.077655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:06.075867    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n"
Oct 30 23:30:06.099797 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:06.090790    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsm5g\" (UniqueName: \"kubernetes.io/projected/4ea6ae1c-d046-499e-89c6-22a626aa98c4-kube-api-access-xsm5g\") pod \"payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn\" (UID: \"4ea6ae1c-d046-499e-89c6-22a626aa98c4\") " pod="openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn"
Oct 30 23:30:06.099797 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:06.091200    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea6ae1c-d046-499e-89c6-22a626aa98c4-plugins-config-volume\") pod \"payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn\" (UID: \"4ea6ae1c-d046-499e-89c6-22a626aa98c4\") " pod="openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn"
Oct 30 23:30:06.168900 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:06.168868    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea6ae1c-d046-499e-89c6-22a626aa98c4-plugins-config-volume\") pod \"payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn\" (UID: \"4ea6ae1c-d046-499e-89c6-22a626aa98c4\") " pod="openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn"
Oct 30 23:30:06.268849 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:06.268072    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsm5g\" (UniqueName: \"kubernetes.io/projected/4ea6ae1c-d046-499e-89c6-22a626aa98c4-kube-api-access-xsm5g\") pod \"payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn\" (UID: \"4ea6ae1c-d046-499e-89c6-22a626aa98c4\") " pod="openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn"
Oct 30 23:30:06.280461 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:06.280422    5078 scope.go:122] "RemoveContainer" containerID="5e846a83bed73217ec37591eb77ec6c4e5106e10713c44214856586fc69e696a"
Oct 30 23:30:06.484185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:06.484159    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn"
Oct 30 23:30:06.630529 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:30:06.630497    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0471065d_e2f3_4d32_83df_90c783556e1d.slice/crio-64799e019bf5d63d52a51efc3d93325273caaac5b784221920af5d74458ce531 WatchSource:0}: Error finding container 64799e019bf5d63d52a51efc3d93325273caaac5b784221920af5d74458ce531: Status 404 returned error can't find the container with id 64799e019bf5d63d52a51efc3d93325273caaac5b784221920af5d74458ce531
Oct 30 23:30:06.631720 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:06.631693    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n"]
Oct 30 23:30:06.786576 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:06.786547    5078 scope.go:122] "RemoveContainer" containerID="b7825ff61bc1879ecfd710f957dccc53bf69bf154594d71fdc60e30795d40121"
Oct 30 23:30:06.786951 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:30:06.786928    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5_odh-ai-gateway-infra(36e85026-f6e7-4cd8-9e1f-44e57104ca7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5" podUID="36e85026-f6e7-4cd8-9e1f-44e57104ca7d"
Oct 30 23:30:06.788360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:06.788342    5078 scope.go:122] "RemoveContainer" containerID="7447e458ccf510e2d2eabb9015fc0222ba463db3ac70634239352d6e6c13a47f"
Oct 30 23:30:06.788724 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:30:06.788694    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv_odh-ai-gateway-infra(b96df090-0d51-4d43-a5ac-ca40af4a29f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv" podUID="b96df090-0d51-4d43-a5ac-ca40af4a29f2"
Oct 30 23:30:06.790089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:06.790071    5078 scope.go:122] "RemoveContainer" containerID="53e68d97178639db300c78712ff82e006cce581027741b58ea7e6ddba9fb2668"
Oct 30 23:30:06.790365 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:30:06.790342    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890050-t9n8k_odh-ai-gateway-infra(b5c2142b-26d1-455e-a6cf-772a7204dbdb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k" podUID="b5c2142b-26d1-455e-a6cf-772a7204dbdb"
Oct 30 23:30:06.791571 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:06.791552    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n" event={"ID":"0471065d-e2f3-4d32-83df-90c783556e1d","Type":"ContainerStarted","Data":"33c69b6b32a704f9fbd7798a33f0a4c80740ea80e90aa238648bf7e461e7102b"}
Oct 30 23:30:06.791661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:06.791608    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n" event={"ID":"0471065d-e2f3-4d32-83df-90c783556e1d","Type":"ContainerStarted","Data":"64799e019bf5d63d52a51efc3d93325273caaac5b784221920af5d74458ce531"}
Oct 30 23:30:06.791714 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:06.791661    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n"
Oct 30 23:30:06.793313 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:06.793299    5078 scope.go:122] "RemoveContainer" containerID="f46d2364fa8c21fcd94f6a5887734ebc7024f88b98a6e256d99069a815da42d7"
Oct 30 23:30:06.793528 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:30:06.793512    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf_odh-ai-gateway-infra(f3260b0f-1a11-4b46-bd73-71df2bb33461)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf" podUID="f3260b0f-1a11-4b46-bd73-71df2bb33461"
Oct 30 23:30:06.878805 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:30:06.878726    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ea6ae1c_d046_499e_89c6_22a626aa98c4.slice/crio-eb81afea1249465e094e3409be6e73e7153220c1bdd9d8e6732b794d29641ba9 WatchSource:0}: Error finding container eb81afea1249465e094e3409be6e73e7153220c1bdd9d8e6732b794d29641ba9: Status 404 returned error can't find the container with id eb81afea1249465e094e3409be6e73e7153220c1bdd9d8e6732b794d29641ba9
Oct 30 23:30:06.880021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:06.879997    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn"]
Oct 30 23:30:07.261698 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:07.261571    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n" podStartSLOduration=2.261557528 podStartE2EDuration="2.261557528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:07.260224357 +0000 UTC m=+1393.148130713" watchObservedRunningTime="2026-10-30 23:30:07.261557528 +0000 UTC m=+1393.149463880"
Oct 30 23:30:07.820503 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:07.820473    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn" event={"ID":"4ea6ae1c-d046-499e-89c6-22a626aa98c4","Type":"ContainerStarted","Data":"de28fe341a7e895b3f183626864c534aa6887603bb6bd9e537f80a3d40b72ff5"}
Oct 30 23:30:07.821036 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:07.820519    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn" event={"ID":"4ea6ae1c-d046-499e-89c6-22a626aa98c4","Type":"ContainerStarted","Data":"eb81afea1249465e094e3409be6e73e7153220c1bdd9d8e6732b794d29641ba9"}
Oct 30 23:30:07.821036 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:07.820553    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn"
Oct 30 23:30:07.937172 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:07.937125    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn" podStartSLOduration=2.937098404 podStartE2EDuration="2.937098404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:07.933373355 +0000 UTC m=+1393.821279708" watchObservedRunningTime="2026-10-30 23:30:07.937098404 +0000 UTC m=+1393.825004756"
Oct 30 23:30:12.452678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.452647    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl"
Oct 30 23:30:12.695996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.695957    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"]
Oct 30 23:30:12.700667 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.700646    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.760964 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.760892    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"]
Oct 30 23:30:12.816319 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.816288    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.816496 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.816343    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4q8r\" (UniqueName: \"kubernetes.io/projected/e0f4988d-873c-469e-80b6-652ab2cadb2b-kube-api-access-l4q8r\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.816496 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.816447    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.816652 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.816629    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.816708 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.816679    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f4988d-873c-469e-80b6-652ab2cadb2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.816860 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.816843    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.816931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.816920    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.817063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.817048    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.817127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.817085    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.918350 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.918317    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.918512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.918384    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.918512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.918473    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.918512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.918506    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.918668 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.918579    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.918668 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.918643    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4q8r\" (UniqueName: \"kubernetes.io/projected/e0f4988d-873c-469e-80b6-652ab2cadb2b-kube-api-access-l4q8r\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.918771 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.918677    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.918771 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.918753    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.918874 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.918784    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f4988d-873c-469e-80b6-652ab2cadb2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.918977 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.918954    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.919079 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.919055    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.919148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.919108    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.919217 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.919197    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.919468 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.919443    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f4988d-873c-469e-80b6-652ab2cadb2b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.921015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.920994    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.921135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.921117    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.976251 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.976174    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4q8r\" (UniqueName: \"kubernetes.io/projected/e0f4988d-873c-469e-80b6-652ab2cadb2b-kube-api-access-l4q8r\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:12.976386 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:12.976250    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-xg4d8\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:13.014270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:13.014238    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:13.460085 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:30:13.460041    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0f4988d_873c_469e_80b6_652ab2cadb2b.slice/crio-e688778a10fd884f0ed3159b72da4e6b852244514269fd1a0658361ba3010194 WatchSource:0}: Error finding container e688778a10fd884f0ed3159b72da4e6b852244514269fd1a0658361ba3010194: Status 404 returned error can't find the container with id e688778a10fd884f0ed3159b72da4e6b852244514269fd1a0658361ba3010194
Oct 30 23:30:13.462566 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:13.462537    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"]
Oct 30 23:30:13.463123 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:13.463094    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:30:13.463229 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:13.463171    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:30:13.463229 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:13.463222    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:30:13.473337 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:13.473310    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6"
Oct 30 23:30:13.473455 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:13.473391    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr"
Oct 30 23:30:13.878553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:13.878515    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8" event={"ID":"e0f4988d-873c-469e-80b6-652ab2cadb2b","Type":"ContainerStarted","Data":"eeffbd0004e46903c0072b3f7fcc01bd3b64fe9e5ea18e64bf95f2b3b809105b"}
Oct 30 23:30:13.878818 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:13.878583    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8" event={"ID":"e0f4988d-873c-469e-80b6-652ab2cadb2b","Type":"ContainerStarted","Data":"e688778a10fd884f0ed3159b72da4e6b852244514269fd1a0658361ba3010194"}
Oct 30 23:30:13.998392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:13.998336    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8" podStartSLOduration=1.998316736 podStartE2EDuration="1.998316736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:13.995624837 +0000 UTC m=+1399.883531191" watchObservedRunningTime="2026-10-30 23:30:13.998316736 +0000 UTC m=+1399.886223091"
Oct 30 23:30:14.014693 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:14.014661    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:14.016539 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:14.016511    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-xg4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 23:30:14.016701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:14.016559    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8" podUID="e0f4988d-873c-469e-80b6-652ab2cadb2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 23:30:15.015682 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:15.015650    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-xg4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 23:30:15.016087 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:15.015709    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8" podUID="e0f4988d-873c-469e-80b6-652ab2cadb2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 23:30:16.015636 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:16.015134    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-xg4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 23:30:16.015636 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:16.015193    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8" podUID="e0f4988d-873c-469e-80b6-652ab2cadb2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 23:30:17.015264 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:17.015227    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-xg4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 23:30:17.015444 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:17.015287    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8" podUID="e0f4988d-873c-469e-80b6-652ab2cadb2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 23:30:17.291514 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:17.291486    5078 scope.go:122] "RemoveContainer" containerID="b7825ff61bc1879ecfd710f957dccc53bf69bf154594d71fdc60e30795d40121"
Oct 30 23:30:17.291931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:17.291645    5078 scope.go:122] "RemoveContainer" containerID="53e68d97178639db300c78712ff82e006cce581027741b58ea7e6ddba9fb2668"
Oct 30 23:30:17.683533 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:30:17.683502    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:30:17.832336 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:17.831107    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n"
Oct 30 23:30:17.930160 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:17.930127    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5" event={"ID":"36e85026-f6e7-4cd8-9e1f-44e57104ca7d","Type":"ContainerStarted","Data":"c67a75535410f53490191e88cd1f1b9f47e7505b3ca57b326e40d38e12d06c24"}
Oct 30 23:30:17.932144 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:17.932120    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k" event={"ID":"b5c2142b-26d1-455e-a6cf-772a7204dbdb","Type":"ContainerStarted","Data":"4bfe1c226b12faa79b4de419a14c81066f9ae4932b2fe50cf467eaaba2a5eecd"}
Oct 30 23:30:18.015007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:18.014930    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-xg4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 23:30:18.015007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:18.014992    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8" podUID="e0f4988d-873c-469e-80b6-652ab2cadb2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 23:30:18.300184 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:18.300150    5078 scope.go:122] "RemoveContainer" containerID="f46d2364fa8c21fcd94f6a5887734ebc7024f88b98a6e256d99069a815da42d7"
Oct 30 23:30:18.300678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:18.300297    5078 scope.go:122] "RemoveContainer" containerID="7447e458ccf510e2d2eabb9015fc0222ba463db3ac70634239352d6e6c13a47f"
Oct 30 23:30:18.995481 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:18.995369    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf" event={"ID":"f3260b0f-1a11-4b46-bd73-71df2bb33461","Type":"ContainerStarted","Data":"fd623aa7ead5d6c7af168d9eb745f01db6892440dc39ba454527916eededc542"}
Oct 30 23:30:18.999407 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:18.999364    5078 generic.go:356] "Generic (PLEG): container finished" podID="36e85026-f6e7-4cd8-9e1f-44e57104ca7d" containerID="c67a75535410f53490191e88cd1f1b9f47e7505b3ca57b326e40d38e12d06c24" exitCode=60
Oct 30 23:30:18.999407 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:18.999390    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5" event={"ID":"36e85026-f6e7-4cd8-9e1f-44e57104ca7d","Type":"ContainerDied","Data":"c67a75535410f53490191e88cd1f1b9f47e7505b3ca57b326e40d38e12d06c24"}
Oct 30 23:30:18.999650 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:18.999460    5078 scope.go:122] "RemoveContainer" containerID="b7825ff61bc1879ecfd710f957dccc53bf69bf154594d71fdc60e30795d40121"
Oct 30 23:30:18.999959 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:18.999940    5078 scope.go:122] "RemoveContainer" containerID="c67a75535410f53490191e88cd1f1b9f47e7505b3ca57b326e40d38e12d06c24"
Oct 30 23:30:19.000365 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:30:19.000336    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5_odh-ai-gateway-infra(36e85026-f6e7-4cd8-9e1f-44e57104ca7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5" podUID="36e85026-f6e7-4cd8-9e1f-44e57104ca7d"
Oct 30 23:30:19.002843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:19.002763    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv" event={"ID":"b96df090-0d51-4d43-a5ac-ca40af4a29f2","Type":"ContainerStarted","Data":"e522956c84500f8f66bcc801beb829ba1c3ca2bd1afab2a2565e065ef59a384a"}
Oct 30 23:30:19.004806 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:19.004788    5078 generic.go:356] "Generic (PLEG): container finished" podID="b5c2142b-26d1-455e-a6cf-772a7204dbdb" containerID="4bfe1c226b12faa79b4de419a14c81066f9ae4932b2fe50cf467eaaba2a5eecd" exitCode=60
Oct 30 23:30:19.005050 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:19.004917    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k" event={"ID":"b5c2142b-26d1-455e-a6cf-772a7204dbdb","Type":"ContainerDied","Data":"4bfe1c226b12faa79b4de419a14c81066f9ae4932b2fe50cf467eaaba2a5eecd"}
Oct 30 23:30:19.005483 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:19.005469    5078 scope.go:122] "RemoveContainer" containerID="4bfe1c226b12faa79b4de419a14c81066f9ae4932b2fe50cf467eaaba2a5eecd"
Oct 30 23:30:19.005854 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:30:19.005838    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890050-t9n8k_odh-ai-gateway-infra(b5c2142b-26d1-455e-a6cf-772a7204dbdb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k" podUID="b5c2142b-26d1-455e-a6cf-772a7204dbdb"
Oct 30 23:30:19.014654 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:19.014640    5078 scope.go:122] "RemoveContainer" containerID="53e68d97178639db300c78712ff82e006cce581027741b58ea7e6ddba9fb2668"
Oct 30 23:30:19.014801 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:19.014776    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-xg4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 23:30:19.014858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:19.014828    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8" podUID="e0f4988d-873c-469e-80b6-652ab2cadb2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 23:30:19.125622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:19.120702    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5"]
Oct 30 23:30:19.234936 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:19.234900    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k"]
Oct 30 23:30:19.291564 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:19.291543    5078 scope.go:122] "RemoveContainer" containerID="d4acd5e9a49d4ac85ca33638373413b69ea0c3e4b62db195f259f2adc5d5b0fb"
Oct 30 23:30:19.865936 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:19.860470    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn"
Oct 30 23:30:20.014578 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.014537    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-xg4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 23:30:20.014781 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.014627    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8" podUID="e0f4988d-873c-469e-80b6-652ab2cadb2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 23:30:20.027639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.024373    5078 generic.go:356] "Generic (PLEG): container finished" podID="f3260b0f-1a11-4b46-bd73-71df2bb33461" containerID="fd623aa7ead5d6c7af168d9eb745f01db6892440dc39ba454527916eededc542" exitCode=60
Oct 30 23:30:20.027639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.024504    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf" event={"ID":"f3260b0f-1a11-4b46-bd73-71df2bb33461","Type":"ContainerDied","Data":"fd623aa7ead5d6c7af168d9eb745f01db6892440dc39ba454527916eededc542"}
Oct 30 23:30:20.027639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.024630    5078 scope.go:122] "RemoveContainer" containerID="f46d2364fa8c21fcd94f6a5887734ebc7024f88b98a6e256d99069a815da42d7"
Oct 30 23:30:20.027639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.025214    5078 scope.go:122] "RemoveContainer" containerID="fd623aa7ead5d6c7af168d9eb745f01db6892440dc39ba454527916eededc542"
Oct 30 23:30:20.027639 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:30:20.025726    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf_odh-ai-gateway-infra(f3260b0f-1a11-4b46-bd73-71df2bb33461)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf" podUID="f3260b0f-1a11-4b46-bd73-71df2bb33461"
Oct 30 23:30:20.028292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.027958    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv"]
Oct 30 23:30:20.033327 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.031579    5078 generic.go:356] "Generic (PLEG): container finished" podID="b96df090-0d51-4d43-a5ac-ca40af4a29f2" containerID="e522956c84500f8f66bcc801beb829ba1c3ca2bd1afab2a2565e065ef59a384a" exitCode=60
Oct 30 23:30:20.042637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.033501    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv" event={"ID":"b96df090-0d51-4d43-a5ac-ca40af4a29f2","Type":"ContainerDied","Data":"e522956c84500f8f66bcc801beb829ba1c3ca2bd1afab2a2565e065ef59a384a"}
Oct 30 23:30:20.042637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.033974    5078 scope.go:122] "RemoveContainer" containerID="e522956c84500f8f66bcc801beb829ba1c3ca2bd1afab2a2565e065ef59a384a"
Oct 30 23:30:20.042637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.034095    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv"
Oct 30 23:30:20.042637 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:30:20.034375    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv_odh-ai-gateway-infra(b96df090-0d51-4d43-a5ac-ca40af4a29f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv" podUID="b96df090-0d51-4d43-a5ac-ca40af4a29f2"
Oct 30 23:30:20.049692 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.048685    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh" event={"ID":"bb6d7732-3d8a-470c-99c1-9bc2fabf83e9","Type":"ContainerStarted","Data":"c52e669ff3833fd92423fb64af99b52708f583d01b155dfff5b4e6b1353649a9"}
Oct 30 23:30:20.088178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.087239    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-40c57b52-dockercfg-6ljlg\""
Oct 30 23:30:20.088178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.087829    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-40c57b52\""
Oct 30 23:30:20.104238 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.104206    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv"]
Oct 30 23:30:20.153958 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.128025    5078 scope.go:122] "RemoveContainer" containerID="7447e458ccf510e2d2eabb9015fc0222ba463db3ac70634239352d6e6c13a47f"
Oct 30 23:30:20.176626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.166772    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf"]
Oct 30 23:30:20.317102 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.296755    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e231029b-bdb1-4433-ba72-0c5471c554a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv\" (UID: \"e231029b-bdb1-4433-ba72-0c5471c554a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv"
Oct 30 23:30:20.317102 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.296903    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztslv\" (UniqueName: \"kubernetes.io/projected/e231029b-bdb1-4433-ba72-0c5471c554a6-kube-api-access-ztslv\") pod \"payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv\" (UID: \"e231029b-bdb1-4433-ba72-0c5471c554a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv"
Oct 30 23:30:20.435866 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.420733    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztslv\" (UniqueName: \"kubernetes.io/projected/e231029b-bdb1-4433-ba72-0c5471c554a6-kube-api-access-ztslv\") pod \"payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv\" (UID: \"e231029b-bdb1-4433-ba72-0c5471c554a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv"
Oct 30 23:30:20.435866 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.421024    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e231029b-bdb1-4433-ba72-0c5471c554a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv\" (UID: \"e231029b-bdb1-4433-ba72-0c5471c554a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv"
Oct 30 23:30:20.435866 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.421942    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e231029b-bdb1-4433-ba72-0c5471c554a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv\" (UID: \"e231029b-bdb1-4433-ba72-0c5471c554a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv"
Oct 30 23:30:20.508197 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.497834    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztslv\" (UniqueName: \"kubernetes.io/projected/e231029b-bdb1-4433-ba72-0c5471c554a6-kube-api-access-ztslv\") pod \"payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv\" (UID: \"e231029b-bdb1-4433-ba72-0c5471c554a6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv"
Oct 30 23:30:20.594010 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.580391    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv"
Oct 30 23:30:20.618921 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.618743    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p"]
Oct 30 23:30:20.633007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.632981    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p"]
Oct 30 23:30:20.633133 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.633103    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p"
Oct 30 23:30:20.640679 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.640648    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv"]
Oct 30 23:30:20.731625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.729833    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382a4000-3d58-4e53-b506-19df51a78e42-plugins-config-volume\") pod \"payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p\" (UID: \"382a4000-3d58-4e53-b506-19df51a78e42\") " pod="openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p"
Oct 30 23:30:20.731625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.729923    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqdfz\" (UniqueName: \"kubernetes.io/projected/382a4000-3d58-4e53-b506-19df51a78e42-kube-api-access-qqdfz\") pod \"payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p\" (UID: \"382a4000-3d58-4e53-b506-19df51a78e42\") " pod="openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p"
Oct 30 23:30:20.757117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.757096    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5"
Oct 30 23:30:20.761674 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.761647    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k"
Oct 30 23:30:20.832453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.832416    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382a4000-3d58-4e53-b506-19df51a78e42-plugins-config-volume\") pod \"payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p\" (UID: \"382a4000-3d58-4e53-b506-19df51a78e42\") " pod="openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p"
Oct 30 23:30:20.832618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.832527    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqdfz\" (UniqueName: \"kubernetes.io/projected/382a4000-3d58-4e53-b506-19df51a78e42-kube-api-access-qqdfz\") pod \"payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p\" (UID: \"382a4000-3d58-4e53-b506-19df51a78e42\") " pod="openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p"
Oct 30 23:30:20.833193 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.833175    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382a4000-3d58-4e53-b506-19df51a78e42-plugins-config-volume\") pod \"payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p\" (UID: \"382a4000-3d58-4e53-b506-19df51a78e42\") " pod="openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p"
Oct 30 23:30:20.890694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.890664    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqdfz\" (UniqueName: \"kubernetes.io/projected/382a4000-3d58-4e53-b506-19df51a78e42-kube-api-access-qqdfz\") pod \"payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p\" (UID: \"382a4000-3d58-4e53-b506-19df51a78e42\") " pod="openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p"
Oct 30 23:30:20.933781 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.933748    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36e85026-f6e7-4cd8-9e1f-44e57104ca7d-kube-api-access-fg97t\" (UniqueName: \"kubernetes.io/projected/36e85026-f6e7-4cd8-9e1f-44e57104ca7d-kube-api-access-fg97t\") pod \"36e85026-f6e7-4cd8-9e1f-44e57104ca7d\" (UID: \"36e85026-f6e7-4cd8-9e1f-44e57104ca7d\") "
Oct 30 23:30:20.933914 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.933882    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36e85026-f6e7-4cd8-9e1f-44e57104ca7d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e85026-f6e7-4cd8-9e1f-44e57104ca7d-maas-api-service-ca\") pod \"36e85026-f6e7-4cd8-9e1f-44e57104ca7d\" (UID: \"36e85026-f6e7-4cd8-9e1f-44e57104ca7d\") "
Oct 30 23:30:20.933969 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.933944    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5c2142b-26d1-455e-a6cf-772a7204dbdb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c2142b-26d1-455e-a6cf-772a7204dbdb-maas-api-service-ca\") pod \"b5c2142b-26d1-455e-a6cf-772a7204dbdb\" (UID: \"b5c2142b-26d1-455e-a6cf-772a7204dbdb\") "
Oct 30 23:30:20.934212 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.934186    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36e85026-f6e7-4cd8-9e1f-44e57104ca7d-maas-api-service-ca" pod "36e85026-f6e7-4cd8-9e1f-44e57104ca7d" (UID: "36e85026-f6e7-4cd8-9e1f-44e57104ca7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:30:20.934212 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.934197    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5c2142b-26d1-455e-a6cf-772a7204dbdb-kube-api-access-d4995\" (UniqueName: \"kubernetes.io/projected/b5c2142b-26d1-455e-a6cf-772a7204dbdb-kube-api-access-d4995\") pod \"b5c2142b-26d1-455e-a6cf-772a7204dbdb\" (UID: \"b5c2142b-26d1-455e-a6cf-772a7204dbdb\") "
Oct 30 23:30:20.934576 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.934235    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c2142b-26d1-455e-a6cf-772a7204dbdb-maas-api-service-ca" pod "b5c2142b-26d1-455e-a6cf-772a7204dbdb" (UID: "b5c2142b-26d1-455e-a6cf-772a7204dbdb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:30:20.934895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.934873    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36e85026-f6e7-4cd8-9e1f-44e57104ca7d-maas-api-service-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:30:20.934977 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.934902    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5c2142b-26d1-455e-a6cf-772a7204dbdb-maas-api-service-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:30:20.936002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.935981    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e85026-f6e7-4cd8-9e1f-44e57104ca7d-kube-api-access-fg97t" pod "36e85026-f6e7-4cd8-9e1f-44e57104ca7d" (UID: "36e85026-f6e7-4cd8-9e1f-44e57104ca7d"). InnerVolumeSpecName "kube-api-access-fg97t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:30:20.936218 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.936196    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c2142b-26d1-455e-a6cf-772a7204dbdb-kube-api-access-d4995" pod "b5c2142b-26d1-455e-a6cf-772a7204dbdb" (UID: "b5c2142b-26d1-455e-a6cf-772a7204dbdb"). InnerVolumeSpecName "kube-api-access-d4995". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:30:20.986948 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:20.986883    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p"
Oct 30 23:30:21.015261 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.015232    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-xg4d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body=
Oct 30 23:30:21.015353 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.015291    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8" podUID="e0f4988d-873c-469e-80b6-652ab2cadb2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused"
Oct 30 23:30:21.037148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.036273    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4995\" (UniqueName: \"kubernetes.io/projected/b5c2142b-26d1-455e-a6cf-772a7204dbdb-kube-api-access-d4995\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:30:21.037148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.036307    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg97t\" (UniqueName: \"kubernetes.io/projected/36e85026-f6e7-4cd8-9e1f-44e57104ca7d-kube-api-access-fg97t\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:30:21.039786 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.039763    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv"]
Oct 30 23:30:21.039956 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:30:21.039936    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode231029b_bdb1_4433_ba72_0c5471c554a6.slice/crio-1cdbedba397662996cd9ace36f8438711e530dc198abe331b33c301123439971 WatchSource:0}: Error finding container 1cdbedba397662996cd9ace36f8438711e530dc198abe331b33c301123439971: Status 404 returned error can't find the container with id 1cdbedba397662996cd9ace36f8438711e530dc198abe331b33c301123439971
Oct 30 23:30:21.078360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.077187    5078 generic.go:356] "Generic (PLEG): container finished" podID="bb6d7732-3d8a-470c-99c1-9bc2fabf83e9" containerID="c52e669ff3833fd92423fb64af99b52708f583d01b155dfff5b4e6b1353649a9" exitCode=60
Oct 30 23:30:21.078360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.077307    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh" event={"ID":"bb6d7732-3d8a-470c-99c1-9bc2fabf83e9","Type":"ContainerDied","Data":"c52e669ff3833fd92423fb64af99b52708f583d01b155dfff5b4e6b1353649a9"}
Oct 30 23:30:21.078360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.077361    5078 scope.go:122] "RemoveContainer" containerID="d4acd5e9a49d4ac85ca33638373413b69ea0c3e4b62db195f259f2adc5d5b0fb"
Oct 30 23:30:21.078360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.077954    5078 scope.go:122] "RemoveContainer" containerID="c52e669ff3833fd92423fb64af99b52708f583d01b155dfff5b4e6b1353649a9"
Oct 30 23:30:21.078681 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:30:21.078436    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh_odh-ai-gateway-infra(bb6d7732-3d8a-470c-99c1-9bc2fabf83e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh" podUID="bb6d7732-3d8a-470c-99c1-9bc2fabf83e9"
Oct 30 23:30:21.083451 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.083410    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv" event={"ID":"e231029b-bdb1-4433-ba72-0c5471c554a6","Type":"ContainerStarted","Data":"1cdbedba397662996cd9ace36f8438711e530dc198abe331b33c301123439971"}
Oct 30 23:30:21.085077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.084950    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5"
Oct 30 23:30:21.085077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.085032    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5" event={"ID":"36e85026-f6e7-4cd8-9e1f-44e57104ca7d","Type":"ContainerDied","Data":"3119de5fa0ff209269f2c866b3147cc0232cefcd51003b3009baf03ec5f71c04"}
Oct 30 23:30:21.095796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.095722    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k"
Oct 30 23:30:21.095796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.095744    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k" event={"ID":"b5c2142b-26d1-455e-a6cf-772a7204dbdb","Type":"ContainerDied","Data":"63bbc8007dc33454ea59a03758abd5f9265b00296cfa2e5ae60277ea80574609"}
Oct 30 23:30:21.104471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.104386    5078 scope.go:122] "RemoveContainer" containerID="c67a75535410f53490191e88cd1f1b9f47e7505b3ca57b326e40d38e12d06c24"
Oct 30 23:30:21.245675 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.245277    5078 scope.go:122] "RemoveContainer" containerID="4bfe1c226b12faa79b4de419a14c81066f9ae4932b2fe50cf467eaaba2a5eecd"
Oct 30 23:30:21.362685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.360391    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh"]
Oct 30 23:30:21.418744 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.418503    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p"]
Oct 30 23:30:21.443721 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.443677    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k"]
Oct 30 23:30:21.451015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.450989    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890050-t9n8k"]
Oct 30 23:30:21.454792 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.454770    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf"
Oct 30 23:30:21.483321 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.483300    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv"
Oct 30 23:30:21.564367 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.564332    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96df090-0d51-4d43-a5ac-ca40af4a29f2-kube-api-access-b5jxc\" (UniqueName: \"kubernetes.io/projected/b96df090-0d51-4d43-a5ac-ca40af4a29f2-kube-api-access-b5jxc\") pod \"b96df090-0d51-4d43-a5ac-ca40af4a29f2\" (UID: \"b96df090-0d51-4d43-a5ac-ca40af4a29f2\") "
Oct 30 23:30:21.564524 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.564452    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3260b0f-1a11-4b46-bd73-71df2bb33461-kube-api-access-p4855\" (UniqueName: \"kubernetes.io/projected/f3260b0f-1a11-4b46-bd73-71df2bb33461-kube-api-access-p4855\") pod \"f3260b0f-1a11-4b46-bd73-71df2bb33461\" (UID: \"f3260b0f-1a11-4b46-bd73-71df2bb33461\") "
Oct 30 23:30:21.564524 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.564516    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96df090-0d51-4d43-a5ac-ca40af4a29f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96df090-0d51-4d43-a5ac-ca40af4a29f2-maas-api-service-ca\") pod \"b96df090-0d51-4d43-a5ac-ca40af4a29f2\" (UID: \"b96df090-0d51-4d43-a5ac-ca40af4a29f2\") "
Oct 30 23:30:21.564700 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.564563    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3260b0f-1a11-4b46-bd73-71df2bb33461-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3260b0f-1a11-4b46-bd73-71df2bb33461-maas-api-service-ca\") pod \"f3260b0f-1a11-4b46-bd73-71df2bb33461\" (UID: \"f3260b0f-1a11-4b46-bd73-71df2bb33461\") "
Oct 30 23:30:21.564947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.564924    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96df090-0d51-4d43-a5ac-ca40af4a29f2-maas-api-service-ca" pod "b96df090-0d51-4d43-a5ac-ca40af4a29f2" (UID: "b96df090-0d51-4d43-a5ac-ca40af4a29f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:30:21.565196 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.565107    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3260b0f-1a11-4b46-bd73-71df2bb33461-maas-api-service-ca" pod "f3260b0f-1a11-4b46-bd73-71df2bb33461" (UID: "f3260b0f-1a11-4b46-bd73-71df2bb33461"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:30:21.565282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.565196    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96df090-0d51-4d43-a5ac-ca40af4a29f2-maas-api-service-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:30:21.566535 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.566514    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96df090-0d51-4d43-a5ac-ca40af4a29f2-kube-api-access-b5jxc" pod "b96df090-0d51-4d43-a5ac-ca40af4a29f2" (UID: "b96df090-0d51-4d43-a5ac-ca40af4a29f2"). InnerVolumeSpecName "kube-api-access-b5jxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:30:21.566642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.566573    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3260b0f-1a11-4b46-bd73-71df2bb33461-kube-api-access-p4855" pod "f3260b0f-1a11-4b46-bd73-71df2bb33461" (UID: "f3260b0f-1a11-4b46-bd73-71df2bb33461"). InnerVolumeSpecName "kube-api-access-p4855". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:30:21.666651 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.666479    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5jxc\" (UniqueName: \"kubernetes.io/projected/b96df090-0d51-4d43-a5ac-ca40af4a29f2-kube-api-access-b5jxc\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:30:21.666651 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.666643    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4855\" (UniqueName: \"kubernetes.io/projected/f3260b0f-1a11-4b46-bd73-71df2bb33461-kube-api-access-p4855\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:30:21.666837 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.666660    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3260b0f-1a11-4b46-bd73-71df2bb33461-maas-api-service-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:30:21.887862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.887821    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5"]
Oct 30 23:30:21.892456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:21.892433    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-3c9613-29890050-88xs5"]
Oct 30 23:30:22.106373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.106346    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv"
Oct 30 23:30:22.106562 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.106379    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv" event={"ID":"b96df090-0d51-4d43-a5ac-ca40af4a29f2","Type":"ContainerDied","Data":"9680939e5a6a948a67fbf9b078672ee136b306a374521db7afe4ec95c28b82b9"}
Oct 30 23:30:22.106562 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.106439    5078 scope.go:122] "RemoveContainer" containerID="e522956c84500f8f66bcc801beb829ba1c3ca2bd1afab2a2565e065ef59a384a"
Oct 30 23:30:22.115818 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.113954    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf" event={"ID":"f3260b0f-1a11-4b46-bd73-71df2bb33461","Type":"ContainerDied","Data":"5d1a15defa8b94495adef4527d3e30f773bd7e2eb66e5b06ab2539e00163a157"}
Oct 30 23:30:22.115818 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.114049    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf"
Oct 30 23:30:22.123062 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.122863    5078 scope.go:122] "RemoveContainer" containerID="fd623aa7ead5d6c7af168d9eb745f01db6892440dc39ba454527916eededc542"
Oct 30 23:30:22.130471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.130440    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv" event={"ID":"e231029b-bdb1-4433-ba72-0c5471c554a6","Type":"ContainerStarted","Data":"2901a5c88b9b15cdf6d373712bd13fd221f6cb3b0fe93fa3e6e1dad3028849e0"}
Oct 30 23:30:22.135632 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.131200    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv"
Oct 30 23:30:22.135715 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.135646    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p" event={"ID":"382a4000-3d58-4e53-b506-19df51a78e42","Type":"ContainerStarted","Data":"be6af15d29595c8b270e0ccc5bfa99502f95049e554b552a06a661923cef171e"}
Oct 30 23:30:22.135715 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.135693    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p" event={"ID":"382a4000-3d58-4e53-b506-19df51a78e42","Type":"ContainerStarted","Data":"3d323acfe559f56dca65fba78c2eda4f422d225b55c68737531f4e16cd506c8c"}
Oct 30 23:30:22.136817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.136792    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p"
Oct 30 23:30:22.334426 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.334394    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36e85026-f6e7-4cd8-9e1f-44e57104ca7d" path="/var/lib/kubelet/pods/36e85026-f6e7-4cd8-9e1f-44e57104ca7d/volumes"
Oct 30 23:30:22.335080 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.335055    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5c2142b-26d1-455e-a6cf-772a7204dbdb" path="/var/lib/kubelet/pods/b5c2142b-26d1-455e-a6cf-772a7204dbdb/volumes"
Oct 30 23:30:22.337562 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.337535    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh"
Oct 30 23:30:22.376915 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.376891    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb6d7732-3d8a-470c-99c1-9bc2fabf83e9-kube-api-access-6x5vs\" (UniqueName: \"kubernetes.io/projected/bb6d7732-3d8a-470c-99c1-9bc2fabf83e9-kube-api-access-6x5vs\") pod \"bb6d7732-3d8a-470c-99c1-9bc2fabf83e9\" (UID: \"bb6d7732-3d8a-470c-99c1-9bc2fabf83e9\") "
Oct 30 23:30:22.377193 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.377173    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb6d7732-3d8a-470c-99c1-9bc2fabf83e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6d7732-3d8a-470c-99c1-9bc2fabf83e9-maas-api-service-ca\") pod \"bb6d7732-3d8a-470c-99c1-9bc2fabf83e9\" (UID: \"bb6d7732-3d8a-470c-99c1-9bc2fabf83e9\") "
Oct 30 23:30:22.377617 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.377564    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb6d7732-3d8a-470c-99c1-9bc2fabf83e9-maas-api-service-ca" pod "bb6d7732-3d8a-470c-99c1-9bc2fabf83e9" (UID: "bb6d7732-3d8a-470c-99c1-9bc2fabf83e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:30:22.377860 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.377838    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb6d7732-3d8a-470c-99c1-9bc2fabf83e9-maas-api-service-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:30:22.379753 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.379721    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb6d7732-3d8a-470c-99c1-9bc2fabf83e9-kube-api-access-6x5vs" pod "bb6d7732-3d8a-470c-99c1-9bc2fabf83e9" (UID: "bb6d7732-3d8a-470c-99c1-9bc2fabf83e9"). InnerVolumeSpecName "kube-api-access-6x5vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:30:22.399972 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.399895    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p" podStartSLOduration=2.3998827990000002 podStartE2EDuration="2.399882799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:22.394098235 +0000 UTC m=+1408.282004589" watchObservedRunningTime="2026-10-30 23:30:22.399882799 +0000 UTC m=+1408.287789152"
Oct 30 23:30:22.479006 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.478974    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x5vs\" (UniqueName: \"kubernetes.io/projected/bb6d7732-3d8a-470c-99c1-9bc2fabf83e9-kube-api-access-6x5vs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:30:22.524631 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.512815    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv"]
Oct 30 23:30:22.524631 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.522202    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-2cfb80-29890050-lv4cv"]
Oct 30 23:30:22.642077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.642018    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv" podStartSLOduration=3.6419971970000002 podStartE2EDuration="3.641997197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:30:22.633381289 +0000 UTC m=+1408.521287658" watchObservedRunningTime="2026-10-30 23:30:22.641997197 +0000 UTC m=+1408.529903550"
Oct 30 23:30:22.800623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.795515    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf"]
Oct 30 23:30:22.829623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.829400    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890050-kg8tf"]
Oct 30 23:30:22.903435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.897877    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:22.914417 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.909762    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:22.920414 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:22.918651    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:30:23.162666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:23.158330    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh" event={"ID":"bb6d7732-3d8a-470c-99c1-9bc2fabf83e9","Type":"ContainerDied","Data":"865d7d53d6a411d6c58bcc07d06be407dca9c7b473a57332f4c9a7556c67f13a"}
Oct 30 23:30:23.162666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:23.158453    5078 scope.go:122] "RemoveContainer" containerID="c52e669ff3833fd92423fb64af99b52708f583d01b155dfff5b4e6b1353649a9"
Oct 30 23:30:23.162666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:23.158676    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh"
Oct 30 23:30:23.330499 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:23.330469    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh"]
Oct 30 23:30:23.335041 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:23.335021    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-989a4f-29890050-577sh"]
Oct 30 23:30:24.300046 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:24.300014    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b96df090-0d51-4d43-a5ac-ca40af4a29f2" path="/var/lib/kubelet/pods/b96df090-0d51-4d43-a5ac-ca40af4a29f2/volumes"
Oct 30 23:30:24.300587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:24.300571    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb6d7732-3d8a-470c-99c1-9bc2fabf83e9" path="/var/lib/kubelet/pods/bb6d7732-3d8a-470c-99c1-9bc2fabf83e9/volumes"
Oct 30 23:30:24.301036 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:24.301022    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3260b0f-1a11-4b46-bd73-71df2bb33461" path="/var/lib/kubelet/pods/f3260b0f-1a11-4b46-bd73-71df2bb33461/volumes"
Oct 30 23:30:34.180604 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:34.180572    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p"
Oct 30 23:30:34.181175 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:34.180846    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv"
Oct 30 23:30:46.554231 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554199    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx"]
Oct 30 23:30:46.554836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554822    5078 cpu_manager.go:414] "containerMap: removing container" podUID="36e85026-f6e7-4cd8-9e1f-44e57104ca7d" containerName="cleanup"
Oct 30 23:30:46.554836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554836    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e85026-f6e7-4cd8-9e1f-44e57104ca7d" containerName="cleanup"
Oct 30 23:30:46.554918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554846    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b96df090-0d51-4d43-a5ac-ca40af4a29f2" containerName="cleanup"
Oct 30 23:30:46.554918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554851    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96df090-0d51-4d43-a5ac-ca40af4a29f2" containerName="cleanup"
Oct 30 23:30:46.554918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554863    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f3260b0f-1a11-4b46-bd73-71df2bb33461" containerName="cleanup"
Oct 30 23:30:46.554918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554867    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3260b0f-1a11-4b46-bd73-71df2bb33461" containerName="cleanup"
Oct 30 23:30:46.554918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554873    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f3260b0f-1a11-4b46-bd73-71df2bb33461" containerName="cleanup"
Oct 30 23:30:46.554918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554877    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3260b0f-1a11-4b46-bd73-71df2bb33461" containerName="cleanup"
Oct 30 23:30:46.554918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554886    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b5c2142b-26d1-455e-a6cf-772a7204dbdb" containerName="cleanup"
Oct 30 23:30:46.554918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554890    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c2142b-26d1-455e-a6cf-772a7204dbdb" containerName="cleanup"
Oct 30 23:30:46.554918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554899    5078 cpu_manager.go:414] "containerMap: removing container" podUID="bb6d7732-3d8a-470c-99c1-9bc2fabf83e9" containerName="cleanup"
Oct 30 23:30:46.554918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554903    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6d7732-3d8a-470c-99c1-9bc2fabf83e9" containerName="cleanup"
Oct 30 23:30:46.554918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554914    5078 cpu_manager.go:414] "containerMap: removing container" podUID="bb6d7732-3d8a-470c-99c1-9bc2fabf83e9" containerName="cleanup"
Oct 30 23:30:46.554918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554918    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6d7732-3d8a-470c-99c1-9bc2fabf83e9" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554929    5078 cpu_manager.go:414] "containerMap: removing container" podUID="bb6d7732-3d8a-470c-99c1-9bc2fabf83e9" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554934    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb6d7732-3d8a-470c-99c1-9bc2fabf83e9" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554941    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f3260b0f-1a11-4b46-bd73-71df2bb33461" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554945    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3260b0f-1a11-4b46-bd73-71df2bb33461" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554959    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b5c2142b-26d1-455e-a6cf-772a7204dbdb" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554964    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c2142b-26d1-455e-a6cf-772a7204dbdb" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554977    5078 cpu_manager.go:414] "containerMap: removing container" podUID="36e85026-f6e7-4cd8-9e1f-44e57104ca7d" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554981    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e85026-f6e7-4cd8-9e1f-44e57104ca7d" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554987    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b5c2142b-26d1-455e-a6cf-772a7204dbdb" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554991    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c2142b-26d1-455e-a6cf-772a7204dbdb" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.554999    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b96df090-0d51-4d43-a5ac-ca40af4a29f2" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.555002    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96df090-0d51-4d43-a5ac-ca40af4a29f2" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.555093    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6d7732-3d8a-470c-99c1-9bc2fabf83e9" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.555102    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6d7732-3d8a-470c-99c1-9bc2fabf83e9" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.555107    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb6d7732-3d8a-470c-99c1-9bc2fabf83e9" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.555114    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3260b0f-1a11-4b46-bd73-71df2bb33461" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.555119    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e85026-f6e7-4cd8-9e1f-44e57104ca7d" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.555124    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c2142b-26d1-455e-a6cf-772a7204dbdb" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.555131    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e85026-f6e7-4cd8-9e1f-44e57104ca7d" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.555135    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c2142b-26d1-455e-a6cf-772a7204dbdb" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.555140    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96df090-0d51-4d43-a5ac-ca40af4a29f2" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.555145    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96df090-0d51-4d43-a5ac-ca40af4a29f2" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.555151    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e85026-f6e7-4cd8-9e1f-44e57104ca7d" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.555156    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96df090-0d51-4d43-a5ac-ca40af4a29f2" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.555161    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3260b0f-1a11-4b46-bd73-71df2bb33461" containerName="cleanup"
Oct 30 23:30:46.555270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.555167    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c2142b-26d1-455e-a6cf-772a7204dbdb" containerName="cleanup"
Oct 30 23:30:46.558970 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.558954    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx"
Oct 30 23:30:46.613850 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.613822    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx"]
Oct 30 23:30:46.657857 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.657823    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:30:46.658039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.657878    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:30:46.658039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.657899    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:30:46.660544 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.660525    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:30:46.661223 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.661201    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5e60c9-45fa-479f-8166-142e02a4ec18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbbb3083-z2hpx\" (UID: \"9f5e60c9-45fa-479f-8166-142e02a4ec18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx"
Oct 30 23:30:46.661268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.661237    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-285tb\" (UniqueName: \"kubernetes.io/projected/9f5e60c9-45fa-479f-8166-142e02a4ec18-kube-api-access-285tb\") pod \"maas-api-revoke-sub-fbbb3083-z2hpx\" (UID: \"9f5e60c9-45fa-479f-8166-142e02a4ec18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx"
Oct 30 23:30:46.661412 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.661397    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5e60c9-45fa-479f-8166-142e02a4ec18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbbb3083-z2hpx\" (UID: \"9f5e60c9-45fa-479f-8166-142e02a4ec18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx"
Oct 30 23:30:46.763263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.763231    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5e60c9-45fa-479f-8166-142e02a4ec18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbbb3083-z2hpx\" (UID: \"9f5e60c9-45fa-479f-8166-142e02a4ec18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx"
Oct 30 23:30:46.763443 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.763423    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5e60c9-45fa-479f-8166-142e02a4ec18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbbb3083-z2hpx\" (UID: \"9f5e60c9-45fa-479f-8166-142e02a4ec18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx"
Oct 30 23:30:46.763507 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.763463    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-285tb\" (UniqueName: \"kubernetes.io/projected/9f5e60c9-45fa-479f-8166-142e02a4ec18-kube-api-access-285tb\") pod \"maas-api-revoke-sub-fbbb3083-z2hpx\" (UID: \"9f5e60c9-45fa-479f-8166-142e02a4ec18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx"
Oct 30 23:30:46.764060 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.764035    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5e60c9-45fa-479f-8166-142e02a4ec18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fbbb3083-z2hpx\" (UID: \"9f5e60c9-45fa-479f-8166-142e02a4ec18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx"
Oct 30 23:30:46.765759 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.765741    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5e60c9-45fa-479f-8166-142e02a4ec18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fbbb3083-z2hpx\" (UID: \"9f5e60c9-45fa-479f-8166-142e02a4ec18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx"
Oct 30 23:30:46.821305 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.821235    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-285tb\" (UniqueName: \"kubernetes.io/projected/9f5e60c9-45fa-479f-8166-142e02a4ec18-kube-api-access-285tb\") pod \"maas-api-revoke-sub-fbbb3083-z2hpx\" (UID: \"9f5e60c9-45fa-479f-8166-142e02a4ec18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx"
Oct 30 23:30:46.872926 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:46.872898    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx"
Oct 30 23:30:47.263493 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:47.263459    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx"]
Oct 30 23:30:47.515465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:47.515384    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx" event={"ID":"9f5e60c9-45fa-479f-8166-142e02a4ec18","Type":"ContainerStarted","Data":"ad6e1dedb332327330b5d472dff3a5e5a49611ec4085e7982631f2d2fb64755f"}
Oct 30 23:30:47.591932 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:30:47.591900    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:30:48.526209 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:48.526125    5078 generic.go:356] "Generic (PLEG): container finished" podID="9f5e60c9-45fa-479f-8166-142e02a4ec18" containerID="4e28b566dbf46ce2c526ea15e9621d214b344da4e8680dbb8bb907f4ee9e9f24" exitCode=0
Oct 30 23:30:48.526365 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:48.526209    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx" event={"ID":"9f5e60c9-45fa-479f-8166-142e02a4ec18","Type":"ContainerDied","Data":"4e28b566dbf46ce2c526ea15e9621d214b344da4e8680dbb8bb907f4ee9e9f24"}
Oct 30 23:30:49.719670 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:49.719645    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx"
Oct 30 23:30:49.826700 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:49.826676    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f5e60c9-45fa-479f-8166-142e02a4ec18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5e60c9-45fa-479f-8166-142e02a4ec18-maas-api-openshift-ca\") pod \"9f5e60c9-45fa-479f-8166-142e02a4ec18\" (UID: \"9f5e60c9-45fa-479f-8166-142e02a4ec18\") "
Oct 30 23:30:49.826881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:49.826718    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f5e60c9-45fa-479f-8166-142e02a4ec18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5e60c9-45fa-479f-8166-142e02a4ec18-maas-api-xks-ca\") pod \"9f5e60c9-45fa-479f-8166-142e02a4ec18\" (UID: \"9f5e60c9-45fa-479f-8166-142e02a4ec18\") "
Oct 30 23:30:49.826881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:49.826800    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f5e60c9-45fa-479f-8166-142e02a4ec18-kube-api-access-285tb\" (UniqueName: \"kubernetes.io/projected/9f5e60c9-45fa-479f-8166-142e02a4ec18-kube-api-access-285tb\") pod \"9f5e60c9-45fa-479f-8166-142e02a4ec18\" (UID: \"9f5e60c9-45fa-479f-8166-142e02a4ec18\") "
Oct 30 23:30:49.827089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:49.827063    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5e60c9-45fa-479f-8166-142e02a4ec18-maas-api-openshift-ca" pod "9f5e60c9-45fa-479f-8166-142e02a4ec18" (UID: "9f5e60c9-45fa-479f-8166-142e02a4ec18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:30:49.827393 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:49.827377    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5e60c9-45fa-479f-8166-142e02a4ec18-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:30:49.829135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:49.829107    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5e60c9-45fa-479f-8166-142e02a4ec18-kube-api-access-285tb" pod "9f5e60c9-45fa-479f-8166-142e02a4ec18" (UID: "9f5e60c9-45fa-479f-8166-142e02a4ec18"). InnerVolumeSpecName "kube-api-access-285tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:30:49.829698 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:49.829671    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f5e60c9-45fa-479f-8166-142e02a4ec18-maas-api-xks-ca" pod "9f5e60c9-45fa-479f-8166-142e02a4ec18" (UID: "9f5e60c9-45fa-479f-8166-142e02a4ec18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:30:49.928658 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:49.928584    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-285tb\" (UniqueName: \"kubernetes.io/projected/9f5e60c9-45fa-479f-8166-142e02a4ec18-kube-api-access-285tb\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:30:49.928658 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:49.928656    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5e60c9-45fa-479f-8166-142e02a4ec18-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:30:50.547293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:50.547262    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx"
Oct 30 23:30:50.547491 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:50.547300    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx" event={"ID":"9f5e60c9-45fa-479f-8166-142e02a4ec18","Type":"ContainerDied","Data":"ad6e1dedb332327330b5d472dff3a5e5a49611ec4085e7982631f2d2fb64755f"}
Oct 30 23:30:50.547491 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:30:50.547347    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad6e1dedb332327330b5d472dff3a5e5a49611ec4085e7982631f2d2fb64755f"
Oct 30 23:31:00.457967 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:00.457934    5078 scope.go:122] "RemoveContainer" containerID="7f7e64bf704c513437d5016a50a4ecaa194eb452f7159e14239ee309490cb858"
Oct 30 23:31:00.468765 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:00.468742    5078 scope.go:122] "RemoveContainer" containerID="f426bfe307cf03a9876f4330514e4ab83354ed8bf962c187eeb6e7dfa2deb14c"
Oct 30 23:31:00.478775 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:00.478747    5078 scope.go:122] "RemoveContainer" containerID="4d536f1667d6314d2675a9c878bd2799b033063651d32b4df96d873fbdea6591"
Oct 30 23:31:00.488712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:00.488693    5078 scope.go:122] "RemoveContainer" containerID="ee8334303a91988aede8266b6442c4e6024f93b4846e5d94cd4fa88c89c08907"
Oct 30 23:31:00.500311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:00.500290    5078 scope.go:122] "RemoveContainer" containerID="e502c06a4701e9004579cf410f9ee7ee6aa54d9db478c9a76882bb40ec5b4c8b"
Oct 30 23:31:00.512118 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:00.512093    5078 scope.go:122] "RemoveContainer" containerID="8d88a4b586da6ad0702e94a33d56a089cf11728c6ae9aebfcd3b4f559836c552"
Oct 30 23:31:00.521121 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:00.521104    5078 scope.go:122] "RemoveContainer" containerID="bd8b8673aa51417e5bad81a40878abd6522b9f499701a7bd7f28fbde4c45942d"
Oct 30 23:31:13.968823 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:13.968794    5078 patch_prober.go:29] interesting pod/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 23:31:13.969177 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:13.968854    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7" podUID="d511dc2f-93c4-41f7-a6a2-3def48b402b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 23:31:14.882925 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:14.882891    5078 generic.go:356] "Generic (PLEG): container finished" podID="d511dc2f-93c4-41f7-a6a2-3def48b402b4" containerID="2235418a3fbb2c50be88a3744a178e7edca02abd8dcb0facab138a7b08c3d9f8" exitCode=0
Oct 30 23:31:14.883098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:14.882960    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7" event={"ID":"d511dc2f-93c4-41f7-a6a2-3def48b402b4","Type":"ContainerDied","Data":"2235418a3fbb2c50be88a3744a178e7edca02abd8dcb0facab138a7b08c3d9f8"}
Oct 30 23:31:14.883433 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:14.883410    5078 scope.go:122] "RemoveContainer" containerID="2235418a3fbb2c50be88a3744a178e7edca02abd8dcb0facab138a7b08c3d9f8"
Oct 30 23:31:14.885047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:14.885017    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:31:14.885170 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:14.885128    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:31:14.885236 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:14.885183    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:31:15.894504 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:15.894471    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7" event={"ID":"d511dc2f-93c4-41f7-a6a2-3def48b402b4","Type":"ContainerStarted","Data":"b088dcff896b0b2048897f9d0ec35a55a9669c71507e4efc5a95061a756094c4"}
Oct 30 23:31:16.232704 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:16.232615    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:31:16.232877 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:16.232761    5078 patch_prober.go:29] interesting pod/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 23:31:16.232877 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:16.232803    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7" podUID="d511dc2f-93c4-41f7-a6a2-3def48b402b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 23:31:17.232825 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:17.232793    5078 patch_prober.go:29] interesting pod/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 23:31:17.233303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:17.232847    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7" podUID="d511dc2f-93c4-41f7-a6a2-3def48b402b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 23:31:17.609762 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:31:17.609728    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:31:18.232949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:18.232914    5078 patch_prober.go:29] interesting pod/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 23:31:18.233332 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:18.232990    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7" podUID="d511dc2f-93c4-41f7-a6a2-3def48b402b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 23:31:19.233465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:19.233433    5078 patch_prober.go:29] interesting pod/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 23:31:19.233894 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:19.233498    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7" podUID="d511dc2f-93c4-41f7-a6a2-3def48b402b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 23:31:20.233610 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:20.233559    5078 patch_prober.go:29] interesting pod/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Oct 30 23:31:20.233999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:20.233651    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7" podUID="d511dc2f-93c4-41f7-a6a2-3def48b402b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Oct 30 23:31:21.237193 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:21.237169    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:31:21.237621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:21.237458    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:31:21.238136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:21.238121    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:31:31.027939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.027898    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l"]
Oct 30 23:31:31.028790 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.028769    5078 cpu_manager.go:414] "containerMap: removing container" podUID="36e85026-f6e7-4cd8-9e1f-44e57104ca7d" containerName="cleanup"
Oct 30 23:31:31.028790 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.028791    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e85026-f6e7-4cd8-9e1f-44e57104ca7d" containerName="cleanup"
Oct 30 23:31:31.028960 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.028804    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b96df090-0d51-4d43-a5ac-ca40af4a29f2" containerName="cleanup"
Oct 30 23:31:31.028960 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.028812    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96df090-0d51-4d43-a5ac-ca40af4a29f2" containerName="cleanup"
Oct 30 23:31:31.028960 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.028824    5078 cpu_manager.go:414] "containerMap: removing container" podUID="9f5e60c9-45fa-479f-8166-142e02a4ec18" containerName="revoke-keys"
Oct 30 23:31:31.028960 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.028832    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5e60c9-45fa-479f-8166-142e02a4ec18" containerName="revoke-keys"
Oct 30 23:31:31.029154 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.029005    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3260b0f-1a11-4b46-bd73-71df2bb33461" containerName="cleanup"
Oct 30 23:31:31.029154 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.029032    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f5e60c9-45fa-479f-8166-142e02a4ec18" containerName="revoke-keys"
Oct 30 23:31:31.032878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.032857    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l"
Oct 30 23:31:31.102091 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.102060    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l"]
Oct 30 23:31:31.127986 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.127198    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:31:31.127986 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.127326    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:31:31.127986 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.127399    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:31:31.138542 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.138521    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:31:31.177278 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.176763    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz4k7\" (UniqueName: \"kubernetes.io/projected/d481549e-d6a4-44a7-ac85-98c466dbab14-kube-api-access-lz4k7\") pod \"maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l\" (UID: \"d481549e-d6a4-44a7-ac85-98c466dbab14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l"
Oct 30 23:31:31.177278 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.176828    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d481549e-d6a4-44a7-ac85-98c466dbab14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l\" (UID: \"d481549e-d6a4-44a7-ac85-98c466dbab14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l"
Oct 30 23:31:31.177278 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.177027    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d481549e-d6a4-44a7-ac85-98c466dbab14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l\" (UID: \"d481549e-d6a4-44a7-ac85-98c466dbab14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l"
Oct 30 23:31:31.279106 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.279014    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d481549e-d6a4-44a7-ac85-98c466dbab14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l\" (UID: \"d481549e-d6a4-44a7-ac85-98c466dbab14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l"
Oct 30 23:31:31.279295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.279232    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz4k7\" (UniqueName: \"kubernetes.io/projected/d481549e-d6a4-44a7-ac85-98c466dbab14-kube-api-access-lz4k7\") pod \"maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l\" (UID: \"d481549e-d6a4-44a7-ac85-98c466dbab14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l"
Oct 30 23:31:31.279295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.279282    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d481549e-d6a4-44a7-ac85-98c466dbab14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l\" (UID: \"d481549e-d6a4-44a7-ac85-98c466dbab14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l"
Oct 30 23:31:31.279935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.279905    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d481549e-d6a4-44a7-ac85-98c466dbab14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l\" (UID: \"d481549e-d6a4-44a7-ac85-98c466dbab14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l"
Oct 30 23:31:31.282484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.282458    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d481549e-d6a4-44a7-ac85-98c466dbab14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l\" (UID: \"d481549e-d6a4-44a7-ac85-98c466dbab14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l"
Oct 30 23:31:31.336111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.336080    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz4k7\" (UniqueName: \"kubernetes.io/projected/d481549e-d6a4-44a7-ac85-98c466dbab14-kube-api-access-lz4k7\") pod \"maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l\" (UID: \"d481549e-d6a4-44a7-ac85-98c466dbab14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l"
Oct 30 23:31:31.347136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.347112    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l"
Oct 30 23:31:31.743787 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:31:31.743751    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd481549e_d6a4_44a7_ac85_98c466dbab14.slice/crio-d6f158f63f49ff789379495725fd9714f5e23a1c16d93f459285882e33d1069e WatchSource:0}: Error finding container d6f158f63f49ff789379495725fd9714f5e23a1c16d93f459285882e33d1069e: Status 404 returned error can't find the container with id d6f158f63f49ff789379495725fd9714f5e23a1c16d93f459285882e33d1069e
Oct 30 23:31:31.745627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:31.745565    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l"]
Oct 30 23:31:32.081309 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:32.081279    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l" event={"ID":"d481549e-d6a4-44a7-ac85-98c466dbab14","Type":"ContainerStarted","Data":"d6f158f63f49ff789379495725fd9714f5e23a1c16d93f459285882e33d1069e"}
Oct 30 23:31:33.090270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:33.090232    5078 generic.go:356] "Generic (PLEG): container finished" podID="d481549e-d6a4-44a7-ac85-98c466dbab14" containerID="0489e15eee2000515fe03aa275baa11de35bf0ce176888156f7d5307790321f7" exitCode=0
Oct 30 23:31:33.090713 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:33.090289    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l" event={"ID":"d481549e-d6a4-44a7-ac85-98c466dbab14","Type":"ContainerDied","Data":"0489e15eee2000515fe03aa275baa11de35bf0ce176888156f7d5307790321f7"}
Oct 30 23:31:34.281846 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:34.281824    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l"
Oct 30 23:31:34.425816 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:34.425739    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d481549e-d6a4-44a7-ac85-98c466dbab14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d481549e-d6a4-44a7-ac85-98c466dbab14-maas-api-xks-ca\") pod \"d481549e-d6a4-44a7-ac85-98c466dbab14\" (UID: \"d481549e-d6a4-44a7-ac85-98c466dbab14\") "
Oct 30 23:31:34.425944 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:34.425917    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d481549e-d6a4-44a7-ac85-98c466dbab14-kube-api-access-lz4k7\" (UniqueName: \"kubernetes.io/projected/d481549e-d6a4-44a7-ac85-98c466dbab14-kube-api-access-lz4k7\") pod \"d481549e-d6a4-44a7-ac85-98c466dbab14\" (UID: \"d481549e-d6a4-44a7-ac85-98c466dbab14\") "
Oct 30 23:31:34.426055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:34.426041    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d481549e-d6a4-44a7-ac85-98c466dbab14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d481549e-d6a4-44a7-ac85-98c466dbab14-maas-api-openshift-ca\") pod \"d481549e-d6a4-44a7-ac85-98c466dbab14\" (UID: \"d481549e-d6a4-44a7-ac85-98c466dbab14\") "
Oct 30 23:31:34.426528 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:34.426429    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d481549e-d6a4-44a7-ac85-98c466dbab14-maas-api-openshift-ca" pod "d481549e-d6a4-44a7-ac85-98c466dbab14" (UID: "d481549e-d6a4-44a7-ac85-98c466dbab14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:31:34.428224 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:34.428196    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d481549e-d6a4-44a7-ac85-98c466dbab14-maas-api-xks-ca" pod "d481549e-d6a4-44a7-ac85-98c466dbab14" (UID: "d481549e-d6a4-44a7-ac85-98c466dbab14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:31:34.428553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:34.428529    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d481549e-d6a4-44a7-ac85-98c466dbab14-kube-api-access-lz4k7" pod "d481549e-d6a4-44a7-ac85-98c466dbab14" (UID: "d481549e-d6a4-44a7-ac85-98c466dbab14"). InnerVolumeSpecName "kube-api-access-lz4k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:31:34.528160 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:34.528127    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d481549e-d6a4-44a7-ac85-98c466dbab14-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:31:34.528160 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:34.528157    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d481549e-d6a4-44a7-ac85-98c466dbab14-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:31:34.528160 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:34.528170    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz4k7\" (UniqueName: \"kubernetes.io/projected/d481549e-d6a4-44a7-ac85-98c466dbab14-kube-api-access-lz4k7\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:31:35.055406 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:35.055373    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f"]
Oct 30 23:31:35.086417 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:35.086391    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e4b5de4-f977f"]
Oct 30 23:31:35.108826 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:35.108804    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l"
Oct 30 23:31:35.108826 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:35.108804    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l" event={"ID":"d481549e-d6a4-44a7-ac85-98c466dbab14","Type":"ContainerDied","Data":"d6f158f63f49ff789379495725fd9714f5e23a1c16d93f459285882e33d1069e"}
Oct 30 23:31:35.109024 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:35.108859    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6f158f63f49ff789379495725fd9714f5e23a1c16d93f459285882e33d1069e"
Oct 30 23:31:36.299989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:36.299949    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10a5cd9-f28f-401e-8199-7a5c022d26b3" path="/var/lib/kubelet/pods/f10a5cd9-f28f-401e-8199-7a5c022d26b3/volumes"
Oct 30 23:31:42.246555 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:42.246522    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm"]
Oct 30 23:31:43.259516 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.259473    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5"]
Oct 30 23:31:43.260216 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.260198    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d481549e-d6a4-44a7-ac85-98c466dbab14" containerName="revoke-keys"
Oct 30 23:31:43.260216 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.260215    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d481549e-d6a4-44a7-ac85-98c466dbab14" containerName="revoke-keys"
Oct 30 23:31:43.260383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.260325    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d481549e-d6a4-44a7-ac85-98c466dbab14" containerName="revoke-keys"
Oct 30 23:31:43.264013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.263992    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5"
Oct 30 23:31:43.316420 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.316393    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:31:43.321269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.321246    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5"]
Oct 30 23:31:43.322162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.322137    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-01c440ca-9twp5\" (UID: \"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5"
Oct 30 23:31:43.322518 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.322392    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-01c440ca-9twp5\" (UID: \"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5"
Oct 30 23:31:43.322633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.322577    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmnph\" (UniqueName: \"kubernetes.io/projected/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-kube-api-access-jmnph\") pod \"maas-api-revoke-keys-e2e-derive-01c440ca-9twp5\" (UID: \"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5"
Oct 30 23:31:43.365119 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.365087    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:31:43.365763 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.365739    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:31:43.365902 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.365789    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:31:43.423574 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.423538    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-01c440ca-9twp5\" (UID: \"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5"
Oct 30 23:31:43.423748 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.423662    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-01c440ca-9twp5\" (UID: \"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5"
Oct 30 23:31:43.423795 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.423761    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmnph\" (UniqueName: \"kubernetes.io/projected/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-kube-api-access-jmnph\") pod \"maas-api-revoke-keys-e2e-derive-01c440ca-9twp5\" (UID: \"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5"
Oct 30 23:31:43.424299 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.424272    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-01c440ca-9twp5\" (UID: \"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5"
Oct 30 23:31:43.426082 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.426062    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-01c440ca-9twp5\" (UID: \"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5"
Oct 30 23:31:43.481267 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.481236    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmnph\" (UniqueName: \"kubernetes.io/projected/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-kube-api-access-jmnph\") pod \"maas-api-revoke-keys-e2e-derive-01c440ca-9twp5\" (UID: \"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5"
Oct 30 23:31:43.587887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:43.587850    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5"
Oct 30 23:31:44.218958 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:31:44.218926    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bbd7ee7_d0e1_4a35_bd59_7bd64b5a9402.slice/crio-0b86336cb67e72270eea440a0681169636ac2f9b456d65ab3947cd0fb4423630 WatchSource:0}: Error finding container 0b86336cb67e72270eea440a0681169636ac2f9b456d65ab3947cd0fb4423630: Status 404 returned error can't find the container with id 0b86336cb67e72270eea440a0681169636ac2f9b456d65ab3947cd0fb4423630
Oct 30 23:31:44.219253 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:44.219231    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5"]
Oct 30 23:31:45.215226 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:45.215191    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5" event={"ID":"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402","Type":"ContainerStarted","Data":"5fb2a6e8c520a4a95a9c26caf7df2f8c02df73fc20713aa48a10261bb8e61c07"}
Oct 30 23:31:45.215686 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:45.215254    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5" event={"ID":"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402","Type":"ContainerStarted","Data":"0b86336cb67e72270eea440a0681169636ac2f9b456d65ab3947cd0fb4423630"}
Oct 30 23:31:45.334835 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:45.334787    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5" podStartSLOduration=2.334773269 podStartE2EDuration="2.334773269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:31:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:31:45.33008369 +0000 UTC m=+1491.217990056" watchObservedRunningTime="2026-10-30 23:31:45.334773269 +0000 UTC m=+1491.222679622"
Oct 30 23:31:46.228552 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:46.228520    5078 generic.go:356] "Generic (PLEG): container finished" podID="2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402" containerID="5fb2a6e8c520a4a95a9c26caf7df2f8c02df73fc20713aa48a10261bb8e61c07" exitCode=0
Oct 30 23:31:46.229021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:46.228638    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5" event={"ID":"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402","Type":"ContainerDied","Data":"5fb2a6e8c520a4a95a9c26caf7df2f8c02df73fc20713aa48a10261bb8e61c07"}
Oct 30 23:31:47.247375 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:47.247332    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm" podUID="4eebd647-3459-494d-b28c-cb27866e4551" containerName="payload-pre-processing" containerID="cri-o://8a1a1874e1f5ff1464f2fa9c4bd48bfa13081fce4bc8ccecd22272f6fae63422" gracePeriod=25
Oct 30 23:31:47.413508 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:47.413483    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5"
Oct 30 23:31:47.529574 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:47.529549    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm"
Oct 30 23:31:47.568199 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:47.568167    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-maas-api-xks-ca\") pod \"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402\" (UID: \"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402\") "
Oct 30 23:31:47.568438 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:47.568419    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-kube-api-access-jmnph\" (UniqueName: \"kubernetes.io/projected/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-kube-api-access-jmnph\") pod \"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402\" (UID: \"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402\") "
Oct 30 23:31:47.568562 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:47.568467    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-maas-api-openshift-ca\") pod \"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402\" (UID: \"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402\") "
Oct 30 23:31:47.569418 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:47.569361    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-maas-api-openshift-ca" pod "2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402" (UID: "2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:31:47.570683 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:47.570657    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-kube-api-access-jmnph" pod "2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402" (UID: "2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402"). InnerVolumeSpecName "kube-api-access-jmnph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:31:47.570798 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:47.570777    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-maas-api-xks-ca" pod "2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402" (UID: "2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:31:47.600822 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:31:47.600785    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:31:47.670170 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:47.670137    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eebd647-3459-494d-b28c-cb27866e4551-kube-api-access-9kk8c\" (UniqueName: \"kubernetes.io/projected/4eebd647-3459-494d-b28c-cb27866e4551-kube-api-access-9kk8c\") pod \"4eebd647-3459-494d-b28c-cb27866e4551\" (UID: \"4eebd647-3459-494d-b28c-cb27866e4551\") "
Oct 30 23:31:47.670373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:47.670282    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eebd647-3459-494d-b28c-cb27866e4551-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eebd647-3459-494d-b28c-cb27866e4551-plugins-config-volume\") pod \"4eebd647-3459-494d-b28c-cb27866e4551\" (UID: \"4eebd647-3459-494d-b28c-cb27866e4551\") "
Oct 30 23:31:47.670689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:47.670663    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eebd647-3459-494d-b28c-cb27866e4551-plugins-config-volume" pod "4eebd647-3459-494d-b28c-cb27866e4551" (UID: "4eebd647-3459-494d-b28c-cb27866e4551"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:31:47.671000 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:47.670981    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eebd647-3459-494d-b28c-cb27866e4551-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:31:47.671074 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:47.671007    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:31:47.671074 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:47.671022    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmnph\" (UniqueName: \"kubernetes.io/projected/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-kube-api-access-jmnph\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:31:47.671074 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:47.671034    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:31:47.673012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:47.672985    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eebd647-3459-494d-b28c-cb27866e4551-kube-api-access-9kk8c" pod "4eebd647-3459-494d-b28c-cb27866e4551" (UID: "4eebd647-3459-494d-b28c-cb27866e4551"). InnerVolumeSpecName "kube-api-access-9kk8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:31:47.772379 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:47.772296    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kk8c\" (UniqueName: \"kubernetes.io/projected/4eebd647-3459-494d-b28c-cb27866e4551-kube-api-access-9kk8c\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:31:48.247720 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:48.247691    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5"
Oct 30 23:31:48.248204 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:48.247728    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5" event={"ID":"2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402","Type":"ContainerDied","Data":"0b86336cb67e72270eea440a0681169636ac2f9b456d65ab3947cd0fb4423630"}
Oct 30 23:31:48.248204 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:48.247775    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b86336cb67e72270eea440a0681169636ac2f9b456d65ab3947cd0fb4423630"
Oct 30 23:31:48.249316 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:48.249292    5078 generic.go:356] "Generic (PLEG): container finished" podID="4eebd647-3459-494d-b28c-cb27866e4551" containerID="8a1a1874e1f5ff1464f2fa9c4bd48bfa13081fce4bc8ccecd22272f6fae63422" exitCode=0
Oct 30 23:31:48.249437 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:48.249329    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm"
Oct 30 23:31:48.249437 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:48.249334    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm" event={"ID":"4eebd647-3459-494d-b28c-cb27866e4551","Type":"ContainerDied","Data":"8a1a1874e1f5ff1464f2fa9c4bd48bfa13081fce4bc8ccecd22272f6fae63422"}
Oct 30 23:31:48.249437 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:48.249363    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm" event={"ID":"4eebd647-3459-494d-b28c-cb27866e4551","Type":"ContainerDied","Data":"525a0f7480edfd458e75d1f77d82e6895cabec4b32e51503ed383d0725ce7695"}
Oct 30 23:31:48.249437 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:48.249384    5078 scope.go:122] "RemoveContainer" containerID="8a1a1874e1f5ff1464f2fa9c4bd48bfa13081fce4bc8ccecd22272f6fae63422"
Oct 30 23:31:48.263486 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:48.263461    5078 scope.go:122] "RemoveContainer" containerID="8a1a1874e1f5ff1464f2fa9c4bd48bfa13081fce4bc8ccecd22272f6fae63422"
Oct 30 23:31:48.263815 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:31:48.263798    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a1a1874e1f5ff1464f2fa9c4bd48bfa13081fce4bc8ccecd22272f6fae63422\": container with ID starting with 8a1a1874e1f5ff1464f2fa9c4bd48bfa13081fce4bc8ccecd22272f6fae63422 not found: ID does not exist" containerID="8a1a1874e1f5ff1464f2fa9c4bd48bfa13081fce4bc8ccecd22272f6fae63422"
Oct 30 23:31:48.263897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:48.263823    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a1a1874e1f5ff1464f2fa9c4bd48bfa13081fce4bc8ccecd22272f6fae63422"} err="failed to get container status \"8a1a1874e1f5ff1464f2fa9c4bd48bfa13081fce4bc8ccecd22272f6fae63422\": rpc error: code = NotFound desc = could not find container \"8a1a1874e1f5ff1464f2fa9c4bd48bfa13081fce4bc8ccecd22272f6fae63422\": container with ID starting with 8a1a1874e1f5ff1464f2fa9c4bd48bfa13081fce4bc8ccecd22272f6fae63422 not found: ID does not exist"
Oct 30 23:31:48.418126 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:48.418089    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm"]
Oct 30 23:31:48.422540 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:48.422520    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-49d07c83-586d495bdc-s8nsm"]
Oct 30 23:31:49.060894 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:49.060858    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh"]
Oct 30 23:31:49.102297 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:49.102269    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a142cf1c-sr5gh"]
Oct 30 23:31:50.130261 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.130231    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95"]
Oct 30 23:31:50.130945 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.130929    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4eebd647-3459-494d-b28c-cb27866e4551" containerName="payload-pre-processing"
Oct 30 23:31:50.130999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.130946    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eebd647-3459-494d-b28c-cb27866e4551" containerName="payload-pre-processing"
Oct 30 23:31:50.130999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.130960    5078 cpu_manager.go:414] "containerMap: removing container" podUID="2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402" containerName="revoke-keys"
Oct 30 23:31:50.130999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.130965    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402" containerName="revoke-keys"
Oct 30 23:31:50.131098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.131058    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eebd647-3459-494d-b28c-cb27866e4551" containerName="payload-pre-processing"
Oct 30 23:31:50.131098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.131069    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402" containerName="revoke-keys"
Oct 30 23:31:50.136090 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.136071    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95"
Oct 30 23:31:50.193132 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.193107    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95"]
Oct 30 23:31:50.234939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.234904    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:31:50.234939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.234941    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:31:50.235153 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.234968    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:31:50.238811 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.238783    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:31:50.299281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.299254    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eebd647-3459-494d-b28c-cb27866e4551" path="/var/lib/kubelet/pods/4eebd647-3459-494d-b28c-cb27866e4551/volumes"
Oct 30 23:31:50.299761 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.299746    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a03b6209-bb59-48eb-9521-0e7a65796e65" path="/var/lib/kubelet/pods/a03b6209-bb59-48eb-9521-0e7a65796e65/volumes"
Oct 30 23:31:50.306242 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.306221    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkkt2\" (UniqueName: \"kubernetes.io/projected/8ea74946-96bd-42de-8c06-ea230836fdd3-kube-api-access-pkkt2\") pod \"maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95\" (UID: \"8ea74946-96bd-42de-8c06-ea230836fdd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95"
Oct 30 23:31:50.306329 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.306262    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea74946-96bd-42de-8c06-ea230836fdd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95\" (UID: \"8ea74946-96bd-42de-8c06-ea230836fdd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95"
Oct 30 23:31:50.306524 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.306507    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea74946-96bd-42de-8c06-ea230836fdd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95\" (UID: \"8ea74946-96bd-42de-8c06-ea230836fdd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95"
Oct 30 23:31:50.408097 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.408019    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea74946-96bd-42de-8c06-ea230836fdd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95\" (UID: \"8ea74946-96bd-42de-8c06-ea230836fdd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95"
Oct 30 23:31:50.408097 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.408076    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkkt2\" (UniqueName: \"kubernetes.io/projected/8ea74946-96bd-42de-8c06-ea230836fdd3-kube-api-access-pkkt2\") pod \"maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95\" (UID: \"8ea74946-96bd-42de-8c06-ea230836fdd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95"
Oct 30 23:31:50.408302 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.408123    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea74946-96bd-42de-8c06-ea230836fdd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95\" (UID: \"8ea74946-96bd-42de-8c06-ea230836fdd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95"
Oct 30 23:31:50.408849 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.408818    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea74946-96bd-42de-8c06-ea230836fdd3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95\" (UID: \"8ea74946-96bd-42de-8c06-ea230836fdd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95"
Oct 30 23:31:50.410517 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.410500    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea74946-96bd-42de-8c06-ea230836fdd3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95\" (UID: \"8ea74946-96bd-42de-8c06-ea230836fdd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95"
Oct 30 23:31:50.464503 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.464478    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkkt2\" (UniqueName: \"kubernetes.io/projected/8ea74946-96bd-42de-8c06-ea230836fdd3-kube-api-access-pkkt2\") pod \"maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95\" (UID: \"8ea74946-96bd-42de-8c06-ea230836fdd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95"
Oct 30 23:31:50.749776 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:50.749705    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95"
Oct 30 23:31:51.144064 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:31:51.144033    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea74946_96bd_42de_8c06_ea230836fdd3.slice/crio-ee28e4b9afcaef52e7bed38b47886b258430fd2deb589760528f2cbaeed5679d WatchSource:0}: Error finding container ee28e4b9afcaef52e7bed38b47886b258430fd2deb589760528f2cbaeed5679d: Status 404 returned error can't find the container with id ee28e4b9afcaef52e7bed38b47886b258430fd2deb589760528f2cbaeed5679d
Oct 30 23:31:51.145549 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:51.145523    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95"]
Oct 30 23:31:51.274326 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:51.274290    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95" event={"ID":"8ea74946-96bd-42de-8c06-ea230836fdd3","Type":"ContainerStarted","Data":"ee28e4b9afcaef52e7bed38b47886b258430fd2deb589760528f2cbaeed5679d"}
Oct 30 23:31:52.284204 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:52.284126    5078 generic.go:356] "Generic (PLEG): container finished" podID="8ea74946-96bd-42de-8c06-ea230836fdd3" containerID="6da0a13fa30a5ef34b885b4ee94f865e191b12b6067ec26d6f861283a95f85ac" exitCode=0
Oct 30 23:31:52.284566 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:52.284214    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95" event={"ID":"8ea74946-96bd-42de-8c06-ea230836fdd3","Type":"ContainerDied","Data":"6da0a13fa30a5ef34b885b4ee94f865e191b12b6067ec26d6f861283a95f85ac"}
Oct 30 23:31:53.467494 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:53.467471    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95"
Oct 30 23:31:53.572293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:53.572216    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ea74946-96bd-42de-8c06-ea230836fdd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea74946-96bd-42de-8c06-ea230836fdd3-maas-api-xks-ca\") pod \"8ea74946-96bd-42de-8c06-ea230836fdd3\" (UID: \"8ea74946-96bd-42de-8c06-ea230836fdd3\") "
Oct 30 23:31:53.572423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:53.572409    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea74946-96bd-42de-8c06-ea230836fdd3-kube-api-access-pkkt2\" (UniqueName: \"kubernetes.io/projected/8ea74946-96bd-42de-8c06-ea230836fdd3-kube-api-access-pkkt2\") pod \"8ea74946-96bd-42de-8c06-ea230836fdd3\" (UID: \"8ea74946-96bd-42de-8c06-ea230836fdd3\") "
Oct 30 23:31:53.572473 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:53.572440    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea74946-96bd-42de-8c06-ea230836fdd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea74946-96bd-42de-8c06-ea230836fdd3-maas-api-openshift-ca\") pod \"8ea74946-96bd-42de-8c06-ea230836fdd3\" (UID: \"8ea74946-96bd-42de-8c06-ea230836fdd3\") "
Oct 30 23:31:53.572869 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:53.572843    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea74946-96bd-42de-8c06-ea230836fdd3-maas-api-openshift-ca" pod "8ea74946-96bd-42de-8c06-ea230836fdd3" (UID: "8ea74946-96bd-42de-8c06-ea230836fdd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:31:53.574385 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:53.574363    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea74946-96bd-42de-8c06-ea230836fdd3-maas-api-xks-ca" pod "8ea74946-96bd-42de-8c06-ea230836fdd3" (UID: "8ea74946-96bd-42de-8c06-ea230836fdd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:31:53.574672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:53.574652    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea74946-96bd-42de-8c06-ea230836fdd3-kube-api-access-pkkt2" pod "8ea74946-96bd-42de-8c06-ea230836fdd3" (UID: "8ea74946-96bd-42de-8c06-ea230836fdd3"). InnerVolumeSpecName "kube-api-access-pkkt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:31:53.674580 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:53.674538    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea74946-96bd-42de-8c06-ea230836fdd3-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:31:53.674580 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:53.674579    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkkt2\" (UniqueName: \"kubernetes.io/projected/8ea74946-96bd-42de-8c06-ea230836fdd3-kube-api-access-pkkt2\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:31:53.674580 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:53.674590    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea74946-96bd-42de-8c06-ea230836fdd3-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:31:54.304043 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:54.304015    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95"
Oct 30 23:31:54.304043 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:54.304034    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95" event={"ID":"8ea74946-96bd-42de-8c06-ea230836fdd3","Type":"ContainerDied","Data":"ee28e4b9afcaef52e7bed38b47886b258430fd2deb589760528f2cbaeed5679d"}
Oct 30 23:31:54.304297 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:54.304075    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee28e4b9afcaef52e7bed38b47886b258430fd2deb589760528f2cbaeed5679d"
Oct 30 23:31:54.749628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:54.749543    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr"]
Oct 30 23:31:54.889618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:54.889050    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl"]
Oct 30 23:31:58.518196 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.518163    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"]
Oct 30 23:31:58.518813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.518798    5078 cpu_manager.go:414] "containerMap: removing container" podUID="8ea74946-96bd-42de-8c06-ea230836fdd3" containerName="revoke-keys"
Oct 30 23:31:58.518813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.518813    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea74946-96bd-42de-8c06-ea230836fdd3" containerName="revoke-keys"
Oct 30 23:31:58.519807 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.519780    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea74946-96bd-42de-8c06-ea230836fdd3" containerName="revoke-keys"
Oct 30 23:31:58.530250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.530223    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.567301 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.567275    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"]
Oct 30 23:31:58.571776 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.571757    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.619357 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.619325    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"]
Oct 30 23:31:58.631459 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.631433    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.631664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.631477    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.631664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.631511    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.631782 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.631750    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.631846 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.631829    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.631909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.631868    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw9hl\" (UniqueName: \"kubernetes.io/projected/6ee01da3-c124-4891-bf71-d6f33f50de56-kube-api-access-tw9hl\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.631977 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.631960    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.632017 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.632002    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.632108 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.632087    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.632196 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.632180    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.632260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.632240    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.632316 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.632272    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.632399 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.632385    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.632442 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.632422    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee01da3-c124-4891-bf71-d6f33f50de56-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.632528 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.632513    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fqh8\" (UniqueName: \"kubernetes.io/projected/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-kube-api-access-2fqh8\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.632588 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.632565    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.632693 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.632675    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.632753 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.632714    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.673338 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.673307    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"]
Oct 30 23:31:58.733956 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.733926    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fqh8\" (UniqueName: \"kubernetes.io/projected/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-kube-api-access-2fqh8\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.734114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.733971    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.734114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734007    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.734114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734023    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.734114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734071    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.734419 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734252    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.734419 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734297    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.734419 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734384    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.734584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734442    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.734584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734448    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.734584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734453    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.734584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734469    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.734584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734473    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw9hl\" (UniqueName: \"kubernetes.io/projected/6ee01da3-c124-4891-bf71-d6f33f50de56-kube-api-access-tw9hl\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.734584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734526    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.734584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734568    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.734970 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734616    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.734970 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734674    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.734970 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734729    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.734970 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734758    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.734970 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734810    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.734970 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734860    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee01da3-c124-4891-bf71-d6f33f50de56-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.734970 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734915    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.735490 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734993    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.735490 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.735150    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.735490 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.734756    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.735490 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.735209    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.735490 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.735260    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.735970 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.735943    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee01da3-c124-4891-bf71-d6f33f50de56-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.737030 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.737010    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.737636 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.737584    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.737636 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.737616    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.738301 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.738283    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.806209 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.792043    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.806209 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.792479    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.806209 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.792505    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fqh8\" (UniqueName: \"kubernetes.io/projected/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-kube-api-access-2fqh8\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nbqlq\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:58.806209 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.792683    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw9hl\" (UniqueName: \"kubernetes.io/projected/6ee01da3-c124-4891-bf71-d6f33f50de56-kube-api-access-tw9hl\") pod \"maas-default-gateway-openshift-default-5df48fcc99-m5q59\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.844610 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.844565    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:58.886067 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:58.886039    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:59.280521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:59.280448    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"]
Oct 30 23:31:59.317065 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:31:59.317041    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee01da3_c124_4891_bf71_d6f33f50de56.slice/crio-14edee018a1f6e572ed5549f337f5902244e9921a28aae5efd8ab6e4544efa22 WatchSource:0}: Error finding container 14edee018a1f6e572ed5549f337f5902244e9921a28aae5efd8ab6e4544efa22: Status 404 returned error can't find the container with id 14edee018a1f6e572ed5549f337f5902244e9921a28aae5efd8ab6e4544efa22
Oct 30 23:31:59.319320 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:59.319289    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:31:59.319417 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:59.319370    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:31:59.319417 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:59.319401    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:31:59.331782 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:59.331757    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"]
Oct 30 23:31:59.332814 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:31:59.332790    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd9c44ad_4eb5_47bd_946b_5274d535dfb8.slice/crio-abf9bdc2a4a588ac91051dff8ef5540af8f6064e5b6211be80449669514866d2 WatchSource:0}: Error finding container abf9bdc2a4a588ac91051dff8ef5540af8f6064e5b6211be80449669514866d2: Status 404 returned error can't find the container with id abf9bdc2a4a588ac91051dff8ef5540af8f6064e5b6211be80449669514866d2
Oct 30 23:31:59.335587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:59.335559    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:31:59.335674 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:59.335633    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:31:59.335674 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:59.335663    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:31:59.635163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:59.635126    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq" event={"ID":"fd9c44ad-4eb5-47bd-946b-5274d535dfb8","Type":"ContainerStarted","Data":"1335eb21df951f27911888d6ba67ab358156f07c5df254c33e2add9bfa46b654"}
Oct 30 23:31:59.635736 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:59.635193    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq" event={"ID":"fd9c44ad-4eb5-47bd-946b-5274d535dfb8","Type":"ContainerStarted","Data":"abf9bdc2a4a588ac91051dff8ef5540af8f6064e5b6211be80449669514866d2"}
Oct 30 23:31:59.637540 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:59.637507    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59" event={"ID":"6ee01da3-c124-4891-bf71-d6f33f50de56","Type":"ContainerStarted","Data":"ca2458f8c72a75745c57be4a0d12dd12fa4609ad7bde82037ee762a1e0e236a6"}
Oct 30 23:31:59.637726 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:59.637566    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59" event={"ID":"6ee01da3-c124-4891-bf71-d6f33f50de56","Type":"ContainerStarted","Data":"14edee018a1f6e572ed5549f337f5902244e9921a28aae5efd8ab6e4544efa22"}
Oct 30 23:31:59.750866 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:59.750827    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr" podUID="a68075cb-25c7-41ec-98ef-61ab9535c970" containerName="payload-processing" containerID="cri-o://d3c448ffcff1aefcefe99679ae72dc1150e417b72973fb4742744e33c53b5f9c" gracePeriod=25
Oct 30 23:31:59.845766 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:59.845737    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:31:59.877258 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:59.877211    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq" podStartSLOduration=1.877186249 podStartE2EDuration="1.877186249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:31:59.756700721 +0000 UTC m=+1505.644607076" watchObservedRunningTime="2026-10-30 23:31:59.877186249 +0000 UTC m=+1505.765092601"
Oct 30 23:31:59.880450 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:59.880403    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59" podStartSLOduration=1.880390156 podStartE2EDuration="1.880390156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:31:59.877371862 +0000 UTC m=+1505.765278216" watchObservedRunningTime="2026-10-30 23:31:59.880390156 +0000 UTC m=+1505.768296573"
Oct 30 23:31:59.886671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:59.886580    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:31:59.889728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:31:59.889700    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl" podUID="8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf" containerName="payload-pre-processing" containerID="cri-o://218d63fa936ad92c96ba922783092f2e70ac43108c35cb3d25774ca30259f3c3" gracePeriod=25
Oct 30 23:32:00.037312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.037288    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr"
Oct 30 23:32:00.172581 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.172489    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a68075cb-25c7-41ec-98ef-61ab9535c970-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a68075cb-25c7-41ec-98ef-61ab9535c970-plugins-config-volume\") pod \"a68075cb-25c7-41ec-98ef-61ab9535c970\" (UID: \"a68075cb-25c7-41ec-98ef-61ab9535c970\") "
Oct 30 23:32:00.173002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.172981    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a68075cb-25c7-41ec-98ef-61ab9535c970-kube-api-access-75cmh\" (UniqueName: \"kubernetes.io/projected/a68075cb-25c7-41ec-98ef-61ab9535c970-kube-api-access-75cmh\") pod \"a68075cb-25c7-41ec-98ef-61ab9535c970\" (UID: \"a68075cb-25c7-41ec-98ef-61ab9535c970\") "
Oct 30 23:32:00.205994 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.205928    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a68075cb-25c7-41ec-98ef-61ab9535c970-plugins-config-volume" pod "a68075cb-25c7-41ec-98ef-61ab9535c970" (UID: "a68075cb-25c7-41ec-98ef-61ab9535c970"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:32:00.217579 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.217087    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68075cb-25c7-41ec-98ef-61ab9535c970-kube-api-access-75cmh" pod "a68075cb-25c7-41ec-98ef-61ab9535c970" (UID: "a68075cb-25c7-41ec-98ef-61ab9535c970"). InnerVolumeSpecName "kube-api-access-75cmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:32:00.229720 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.229391    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl"
Oct 30 23:32:00.241725 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.241682    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75cmh\" (UniqueName: \"kubernetes.io/projected/a68075cb-25c7-41ec-98ef-61ab9535c970-kube-api-access-75cmh\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:32:00.241725 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.241718    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a68075cb-25c7-41ec-98ef-61ab9535c970-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:32:00.343063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.343035    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf-kube-api-access-w8rfn\" (UniqueName: \"kubernetes.io/projected/8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf-kube-api-access-w8rfn\") pod \"8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf\" (UID: \"8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf\") "
Oct 30 23:32:00.343216 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.343201    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf-plugins-config-volume\") pod \"8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf\" (UID: \"8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf\") "
Oct 30 23:32:00.343645 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.343590    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf-plugins-config-volume" pod "8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf" (UID: "8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:32:00.343983 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.343969    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:32:00.345266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.345245    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf-kube-api-access-w8rfn" pod "8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf" (UID: "8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf"). InnerVolumeSpecName "kube-api-access-w8rfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:32:00.444863 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.444778    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8rfn\" (UniqueName: \"kubernetes.io/projected/8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf-kube-api-access-w8rfn\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:32:00.646381 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.646349    5078 generic.go:356] "Generic (PLEG): container finished" podID="a68075cb-25c7-41ec-98ef-61ab9535c970" containerID="d3c448ffcff1aefcefe99679ae72dc1150e417b72973fb4742744e33c53b5f9c" exitCode=0
Oct 30 23:32:00.646839 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.646396    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr"
Oct 30 23:32:00.646839 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.646442    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr" event={"ID":"a68075cb-25c7-41ec-98ef-61ab9535c970","Type":"ContainerDied","Data":"d3c448ffcff1aefcefe99679ae72dc1150e417b72973fb4742744e33c53b5f9c"}
Oct 30 23:32:00.646839 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.646499    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr" event={"ID":"a68075cb-25c7-41ec-98ef-61ab9535c970","Type":"ContainerDied","Data":"d4cc25ce259f2ec5b4b048c993a666291c4f6b745db6b08ed34773560e7bb9df"}
Oct 30 23:32:00.646839 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.646533    5078 scope.go:122] "RemoveContainer" containerID="d3c448ffcff1aefcefe99679ae72dc1150e417b72973fb4742744e33c53b5f9c"
Oct 30 23:32:00.648043 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.648002    5078 generic.go:356] "Generic (PLEG): container finished" podID="8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf" containerID="218d63fa936ad92c96ba922783092f2e70ac43108c35cb3d25774ca30259f3c3" exitCode=0
Oct 30 23:32:00.648140 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.648035    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl" event={"ID":"8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf","Type":"ContainerDied","Data":"218d63fa936ad92c96ba922783092f2e70ac43108c35cb3d25774ca30259f3c3"}
Oct 30 23:32:00.648140 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.648050    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl"
Oct 30 23:32:00.648140 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.648087    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl" event={"ID":"8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf","Type":"ContainerDied","Data":"11fe09de98db643edddba24a6f90c06218bc23177648501a6db1939b33a84194"}
Oct 30 23:32:00.702701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.700849    5078 scope.go:122] "RemoveContainer" containerID="d3c448ffcff1aefcefe99679ae72dc1150e417b72973fb4742744e33c53b5f9c"
Oct 30 23:32:00.702701 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:32:00.701280    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3c448ffcff1aefcefe99679ae72dc1150e417b72973fb4742744e33c53b5f9c\": container with ID starting with d3c448ffcff1aefcefe99679ae72dc1150e417b72973fb4742744e33c53b5f9c not found: ID does not exist" containerID="d3c448ffcff1aefcefe99679ae72dc1150e417b72973fb4742744e33c53b5f9c"
Oct 30 23:32:00.702701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.701314    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3c448ffcff1aefcefe99679ae72dc1150e417b72973fb4742744e33c53b5f9c"} err="failed to get container status \"d3c448ffcff1aefcefe99679ae72dc1150e417b72973fb4742744e33c53b5f9c\": rpc error: code = NotFound desc = could not find container \"d3c448ffcff1aefcefe99679ae72dc1150e417b72973fb4742744e33c53b5f9c\": container with ID starting with d3c448ffcff1aefcefe99679ae72dc1150e417b72973fb4742744e33c53b5f9c not found: ID does not exist"
Oct 30 23:32:00.702701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.701337    5078 scope.go:122] "RemoveContainer" containerID="218d63fa936ad92c96ba922783092f2e70ac43108c35cb3d25774ca30259f3c3"
Oct 30 23:32:00.715699 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.715676    5078 scope.go:122] "RemoveContainer" containerID="218d63fa936ad92c96ba922783092f2e70ac43108c35cb3d25774ca30259f3c3"
Oct 30 23:32:00.716035 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:32:00.716008    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"218d63fa936ad92c96ba922783092f2e70ac43108c35cb3d25774ca30259f3c3\": container with ID starting with 218d63fa936ad92c96ba922783092f2e70ac43108c35cb3d25774ca30259f3c3 not found: ID does not exist" containerID="218d63fa936ad92c96ba922783092f2e70ac43108c35cb3d25774ca30259f3c3"
Oct 30 23:32:00.716129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.716041    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"218d63fa936ad92c96ba922783092f2e70ac43108c35cb3d25774ca30259f3c3"} err="failed to get container status \"218d63fa936ad92c96ba922783092f2e70ac43108c35cb3d25774ca30259f3c3\": rpc error: code = NotFound desc = could not find container \"218d63fa936ad92c96ba922783092f2e70ac43108c35cb3d25774ca30259f3c3\": container with ID starting with 218d63fa936ad92c96ba922783092f2e70ac43108c35cb3d25774ca30259f3c3 not found: ID does not exist"
Oct 30 23:32:00.822835 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.822800    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr"]
Oct 30 23:32:00.827694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.827666    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-01c440ca-7786488fc9-jthkr"]
Oct 30 23:32:00.846280 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.846250    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-m5q59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 23:32:00.846403 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.846299    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59" podUID="6ee01da3-c124-4891-bf71-d6f33f50de56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 23:32:00.907775 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.907730    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nbqlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 23:32:00.907973 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.907794    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq" podUID="fd9c44ad-4eb5-47bd-946b-5274d535dfb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 23:32:00.943193 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.943164    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl"]
Oct 30 23:32:00.954061 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:00.954001    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-01c440ca-568df486c9-b85xl"]
Oct 30 23:32:01.845935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:01.845899    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-m5q59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 23:32:01.846348 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:01.845947    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59" podUID="6ee01da3-c124-4891-bf71-d6f33f50de56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 23:32:01.890622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:01.890333    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nbqlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 23:32:01.890622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:01.890394    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq" podUID="fd9c44ad-4eb5-47bd-946b-5274d535dfb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 23:32:02.118157 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:02.118081    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6"]
Oct 30 23:32:02.314748 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:02.314709    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf" path="/var/lib/kubelet/pods/8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf/volumes"
Oct 30 23:32:02.315226 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:02.315206    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a68075cb-25c7-41ec-98ef-61ab9535c970" path="/var/lib/kubelet/pods/a68075cb-25c7-41ec-98ef-61ab9535c970/volumes"
Oct 30 23:32:02.857585 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:02.857546    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-m5q59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 23:32:02.863682 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:02.863645    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59" podUID="6ee01da3-c124-4891-bf71-d6f33f50de56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 23:32:02.914048 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:02.914003    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nbqlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 23:32:02.914355 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:02.914324    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq" podUID="fd9c44ad-4eb5-47bd-946b-5274d535dfb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 23:32:03.881628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:03.879741    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-m5q59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 23:32:03.881628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:03.879799    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59" podUID="6ee01da3-c124-4891-bf71-d6f33f50de56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 23:32:03.903358 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:03.903225    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nbqlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 23:32:03.903358 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:03.903287    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq" podUID="fd9c44ad-4eb5-47bd-946b-5274d535dfb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 23:32:04.845527 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:04.845498    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-m5q59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 23:32:04.845787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:04.845759    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59" podUID="6ee01da3-c124-4891-bf71-d6f33f50de56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 23:32:04.887623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:04.887056    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nbqlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 23:32:04.887623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:04.887133    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq" podUID="fd9c44ad-4eb5-47bd-946b-5274d535dfb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 23:32:05.845674 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:05.845641    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-m5q59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 23:32:05.845847 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:05.845698    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59" podUID="6ee01da3-c124-4891-bf71-d6f33f50de56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 23:32:05.895622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:05.892234    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nbqlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 23:32:05.895622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:05.892289    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq" podUID="fd9c44ad-4eb5-47bd-946b-5274d535dfb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 23:32:06.845884 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:06.845847    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-m5q59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 23:32:06.846096 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:06.845926    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59" podUID="6ee01da3-c124-4891-bf71-d6f33f50de56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 23:32:06.887384 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:06.887348    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nbqlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 23:32:06.887576 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:06.887412    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq" podUID="fd9c44ad-4eb5-47bd-946b-5274d535dfb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 23:32:07.119804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.119715    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6" podUID="ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608" containerName="payload-processing" containerID="cri-o://68bf0181b17594db932b42af3e66a7f476a227a9a5ee07d8fa6f37ff125bd9b1" gracePeriod=25
Oct 30 23:32:07.437727 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.437701    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6"
Oct 30 23:32:07.455164 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.455140    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608-plugins-config-volume\") pod \"ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608\" (UID: \"ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608\") "
Oct 30 23:32:07.455313 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.455186    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608-kube-api-access-wcvcp\" (UniqueName: \"kubernetes.io/projected/ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608-kube-api-access-wcvcp\") pod \"ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608\" (UID: \"ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608\") "
Oct 30 23:32:07.455582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.455558    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608-plugins-config-volume" pod "ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608" (UID: "ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:32:07.456101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.456073    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:32:07.457466 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.457443    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608-kube-api-access-wcvcp" pod "ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608" (UID: "ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608"). InnerVolumeSpecName "kube-api-access-wcvcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:32:07.557689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.557646    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcvcp\" (UniqueName: \"kubernetes.io/projected/ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608-kube-api-access-wcvcp\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:32:07.845891 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.845850    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-m5q59 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused" start-of-body=
Oct 30 23:32:07.846075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.845915    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59" podUID="6ee01da3-c124-4891-bf71-d6f33f50de56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.140:15021/healthz/ready\": dial tcp 10.133.0.140:15021: connect: connection refused"
Oct 30 23:32:07.887426 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.887382    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nbqlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Oct 30 23:32:07.887680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.887446    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq" podUID="fd9c44ad-4eb5-47bd-946b-5274d535dfb8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Oct 30 23:32:07.967324 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.967292    5078 generic.go:356] "Generic (PLEG): container finished" podID="ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608" containerID="68bf0181b17594db932b42af3e66a7f476a227a9a5ee07d8fa6f37ff125bd9b1" exitCode=0
Oct 30 23:32:07.967522 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.967388    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6"
Oct 30 23:32:07.967522 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.967404    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6" event={"ID":"ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608","Type":"ContainerDied","Data":"68bf0181b17594db932b42af3e66a7f476a227a9a5ee07d8fa6f37ff125bd9b1"}
Oct 30 23:32:07.967522 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.967517    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6" event={"ID":"ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608","Type":"ContainerDied","Data":"ff648774cbe651af2d1656009a396197da45ac76fbfe2cffd5852a5f263142d0"}
Oct 30 23:32:07.967758 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.967562    5078 scope.go:122] "RemoveContainer" containerID="68bf0181b17594db932b42af3e66a7f476a227a9a5ee07d8fa6f37ff125bd9b1"
Oct 30 23:32:07.984524 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.984500    5078 scope.go:122] "RemoveContainer" containerID="68bf0181b17594db932b42af3e66a7f476a227a9a5ee07d8fa6f37ff125bd9b1"
Oct 30 23:32:07.984875 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:32:07.984853    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68bf0181b17594db932b42af3e66a7f476a227a9a5ee07d8fa6f37ff125bd9b1\": container with ID starting with 68bf0181b17594db932b42af3e66a7f476a227a9a5ee07d8fa6f37ff125bd9b1 not found: ID does not exist" containerID="68bf0181b17594db932b42af3e66a7f476a227a9a5ee07d8fa6f37ff125bd9b1"
Oct 30 23:32:07.984983 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:07.984881    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68bf0181b17594db932b42af3e66a7f476a227a9a5ee07d8fa6f37ff125bd9b1"} err="failed to get container status \"68bf0181b17594db932b42af3e66a7f476a227a9a5ee07d8fa6f37ff125bd9b1\": rpc error: code = NotFound desc = could not find container \"68bf0181b17594db932b42af3e66a7f476a227a9a5ee07d8fa6f37ff125bd9b1\": container with ID starting with 68bf0181b17594db932b42af3e66a7f476a227a9a5ee07d8fa6f37ff125bd9b1 not found: ID does not exist"
Oct 30 23:32:08.138953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:08.138856    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6"]
Oct 30 23:32:08.147128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:08.147097    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2b6c82a0-6bd86548c5-d9zh6"]
Oct 30 23:32:08.305411 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:08.305361    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608" path="/var/lib/kubelet/pods/ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608/volumes"
Oct 30 23:32:08.890398 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:08.890369    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:32:08.890959 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:08.890938    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:32:08.892043 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:08.892023    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:32:08.950832 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:08.950783    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:32:08.951437 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:08.951416    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:32:08.952281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:08.952261    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:32:14.051768 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:14.051732    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv"]
Oct 30 23:32:14.100926 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:14.100898    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b509852-l8zcv"]
Oct 30 23:32:14.299196 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:14.299163    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3af4294-205a-4ec2-8114-df49ac3a1bbf" path="/var/lib/kubelet/pods/c3af4294-205a-4ec2-8114-df49ac3a1bbf/volumes"
Oct 30 23:32:17.611387 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:32:17.611351    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:32:18.219039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.219010    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh"]
Oct 30 23:32:18.219707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.219691    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608" containerName="payload-processing"
Oct 30 23:32:18.219707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.219708    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608" containerName="payload-processing"
Oct 30 23:32:18.219829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.219725    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a68075cb-25c7-41ec-98ef-61ab9535c970" containerName="payload-processing"
Oct 30 23:32:18.219829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.219731    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a68075cb-25c7-41ec-98ef-61ab9535c970" containerName="payload-processing"
Oct 30 23:32:18.219829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.219744    5078 cpu_manager.go:414] "containerMap: removing container" podUID="8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf" containerName="payload-pre-processing"
Oct 30 23:32:18.219829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.219749    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf" containerName="payload-pre-processing"
Oct 30 23:32:18.219954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.219842    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed3b0cd5-173f-4bc5-a7bf-c4fecf81e608" containerName="payload-processing"
Oct 30 23:32:18.219954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.219851    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a68075cb-25c7-41ec-98ef-61ab9535c970" containerName="payload-processing"
Oct 30 23:32:18.219954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.219858    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e5e69d6-41d7-4abc-a1c1-4479dfb6cbbf" containerName="payload-pre-processing"
Oct 30 23:32:18.223766 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.223744    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh"
Oct 30 23:32:18.286672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.286636    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh"]
Oct 30 23:32:18.327802 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.327761    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:32:18.328054 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.327854    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:32:18.328183 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.327898    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:32:18.328183 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.328171    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:32:18.392591 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.392563    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155ffa13-2141-4a24-91c5-020772b96b34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-40c57b52-27mkh\" (UID: \"155ffa13-2141-4a24-91c5-020772b96b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh"
Oct 30 23:32:18.392786 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.392767    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155ffa13-2141-4a24-91c5-020772b96b34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-40c57b52-27mkh\" (UID: \"155ffa13-2141-4a24-91c5-020772b96b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh"
Oct 30 23:32:18.392830 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.392814    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2r88\" (UniqueName: \"kubernetes.io/projected/155ffa13-2141-4a24-91c5-020772b96b34-kube-api-access-n2r88\") pod \"maas-api-revoke-keys-e2e-ait-40c57b52-27mkh\" (UID: \"155ffa13-2141-4a24-91c5-020772b96b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh"
Oct 30 23:32:18.494491 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.494414    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155ffa13-2141-4a24-91c5-020772b96b34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-40c57b52-27mkh\" (UID: \"155ffa13-2141-4a24-91c5-020772b96b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh"
Oct 30 23:32:18.494491 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.494471    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2r88\" (UniqueName: \"kubernetes.io/projected/155ffa13-2141-4a24-91c5-020772b96b34-kube-api-access-n2r88\") pod \"maas-api-revoke-keys-e2e-ait-40c57b52-27mkh\" (UID: \"155ffa13-2141-4a24-91c5-020772b96b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh"
Oct 30 23:32:18.494778 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.494711    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155ffa13-2141-4a24-91c5-020772b96b34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-40c57b52-27mkh\" (UID: \"155ffa13-2141-4a24-91c5-020772b96b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh"
Oct 30 23:32:18.495650 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.495582    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155ffa13-2141-4a24-91c5-020772b96b34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-40c57b52-27mkh\" (UID: \"155ffa13-2141-4a24-91c5-020772b96b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh"
Oct 30 23:32:18.505584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.505555    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155ffa13-2141-4a24-91c5-020772b96b34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-40c57b52-27mkh\" (UID: \"155ffa13-2141-4a24-91c5-020772b96b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh"
Oct 30 23:32:18.554417 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.554389    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2r88\" (UniqueName: \"kubernetes.io/projected/155ffa13-2141-4a24-91c5-020772b96b34-kube-api-access-n2r88\") pod \"maas-api-revoke-keys-e2e-ait-40c57b52-27mkh\" (UID: \"155ffa13-2141-4a24-91c5-020772b96b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh"
Oct 30 23:32:18.840002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:18.839956    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh"
Oct 30 23:32:19.250364 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:19.250337    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh"]
Oct 30 23:32:19.251128 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:32:19.251099    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod155ffa13_2141_4a24_91c5_020772b96b34.slice/crio-358a99a78ff3ef7c7e3fa37487168ee66c0e52857b4ad6a25c47c9e283ff116c WatchSource:0}: Error finding container 358a99a78ff3ef7c7e3fa37487168ee66c0e52857b4ad6a25c47c9e283ff116c: Status 404 returned error can't find the container with id 358a99a78ff3ef7c7e3fa37487168ee66c0e52857b4ad6a25c47c9e283ff116c
Oct 30 23:32:19.995014 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:19.994984    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc"]
Oct 30 23:32:19.999424 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:19.999406    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc"
Oct 30 23:32:20.013083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:20.013059    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e3dd03-03e7-42cb-a6b6-3680f039b67b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8de42e8c-ftxzc\" (UID: \"02e3dd03-03e7-42cb-a6b6-3680f039b67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc"
Oct 30 23:32:20.013205 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:20.013116    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e3dd03-03e7-42cb-a6b6-3680f039b67b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8de42e8c-ftxzc\" (UID: \"02e3dd03-03e7-42cb-a6b6-3680f039b67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc"
Oct 30 23:32:20.013267 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:20.013210    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rqlg\" (UniqueName: \"kubernetes.io/projected/02e3dd03-03e7-42cb-a6b6-3680f039b67b-kube-api-access-4rqlg\") pod \"maas-api-revoke-sub-8de42e8c-ftxzc\" (UID: \"02e3dd03-03e7-42cb-a6b6-3680f039b67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc"
Oct 30 23:32:20.057612 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:20.057274    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc"]
Oct 30 23:32:20.114690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:20.114579    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e3dd03-03e7-42cb-a6b6-3680f039b67b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8de42e8c-ftxzc\" (UID: \"02e3dd03-03e7-42cb-a6b6-3680f039b67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc"
Oct 30 23:32:20.115079 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:20.114975    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e3dd03-03e7-42cb-a6b6-3680f039b67b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8de42e8c-ftxzc\" (UID: \"02e3dd03-03e7-42cb-a6b6-3680f039b67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc"
Oct 30 23:32:20.115237 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:20.115062    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rqlg\" (UniqueName: \"kubernetes.io/projected/02e3dd03-03e7-42cb-a6b6-3680f039b67b-kube-api-access-4rqlg\") pod \"maas-api-revoke-sub-8de42e8c-ftxzc\" (UID: \"02e3dd03-03e7-42cb-a6b6-3680f039b67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc"
Oct 30 23:32:20.116961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:20.116934    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e3dd03-03e7-42cb-a6b6-3680f039b67b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8de42e8c-ftxzc\" (UID: \"02e3dd03-03e7-42cb-a6b6-3680f039b67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc"
Oct 30 23:32:20.119761 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:20.119730    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh" event={"ID":"155ffa13-2141-4a24-91c5-020772b96b34","Type":"ContainerStarted","Data":"93d71478d09e496ff215f600d545869e458c38c6520bd5ec5860935af20b8398"}
Oct 30 23:32:20.119932 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:20.119915    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh" event={"ID":"155ffa13-2141-4a24-91c5-020772b96b34","Type":"ContainerStarted","Data":"358a99a78ff3ef7c7e3fa37487168ee66c0e52857b4ad6a25c47c9e283ff116c"}
Oct 30 23:32:20.121343 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:20.121317    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e3dd03-03e7-42cb-a6b6-3680f039b67b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8de42e8c-ftxzc\" (UID: \"02e3dd03-03e7-42cb-a6b6-3680f039b67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc"
Oct 30 23:32:20.184209 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:20.184180    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rqlg\" (UniqueName: \"kubernetes.io/projected/02e3dd03-03e7-42cb-a6b6-3680f039b67b-kube-api-access-4rqlg\") pod \"maas-api-revoke-sub-8de42e8c-ftxzc\" (UID: \"02e3dd03-03e7-42cb-a6b6-3680f039b67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc"
Oct 30 23:32:20.247859 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:20.247758    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh" podStartSLOduration=2.247745305 podStartE2EDuration="2.247745305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:32:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:32:20.244785942 +0000 UTC m=+1526.132692296" watchObservedRunningTime="2026-10-30 23:32:20.247745305 +0000 UTC m=+1526.135651682"
Oct 30 23:32:20.314093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:20.314064    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc"
Oct 30 23:32:20.726152 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:32:20.726119    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e3dd03_03e7_42cb_a6b6_3680f039b67b.slice/crio-7af67d73be96d71396bf01e81158c909ef189574a78074c812d21464292dd4f5 WatchSource:0}: Error finding container 7af67d73be96d71396bf01e81158c909ef189574a78074c812d21464292dd4f5: Status 404 returned error can't find the container with id 7af67d73be96d71396bf01e81158c909ef189574a78074c812d21464292dd4f5
Oct 30 23:32:20.728484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:20.728458    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc"]
Oct 30 23:32:21.129671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:21.129641    5078 generic.go:356] "Generic (PLEG): container finished" podID="155ffa13-2141-4a24-91c5-020772b96b34" containerID="93d71478d09e496ff215f600d545869e458c38c6520bd5ec5860935af20b8398" exitCode=0
Oct 30 23:32:21.130077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:21.129729    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh" event={"ID":"155ffa13-2141-4a24-91c5-020772b96b34","Type":"ContainerDied","Data":"93d71478d09e496ff215f600d545869e458c38c6520bd5ec5860935af20b8398"}
Oct 30 23:32:21.131039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:21.131014    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc" event={"ID":"02e3dd03-03e7-42cb-a6b6-3680f039b67b","Type":"ContainerStarted","Data":"7af67d73be96d71396bf01e81158c909ef189574a78074c812d21464292dd4f5"}
Oct 30 23:32:22.140393 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:22.140307    5078 generic.go:356] "Generic (PLEG): container finished" podID="02e3dd03-03e7-42cb-a6b6-3680f039b67b" containerID="f2b15d9131f271ddab270ed40e9a96b7ad979b8cffb4ed7fe6fbe2518190524c" exitCode=0
Oct 30 23:32:22.140393 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:22.140341    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc" event={"ID":"02e3dd03-03e7-42cb-a6b6-3680f039b67b","Type":"ContainerDied","Data":"f2b15d9131f271ddab270ed40e9a96b7ad979b8cffb4ed7fe6fbe2518190524c"}
Oct 30 23:32:22.312351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:22.312329    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh"
Oct 30 23:32:22.391083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:22.390993    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/155ffa13-2141-4a24-91c5-020772b96b34-kube-api-access-n2r88\" (UniqueName: \"kubernetes.io/projected/155ffa13-2141-4a24-91c5-020772b96b34-kube-api-access-n2r88\") pod \"155ffa13-2141-4a24-91c5-020772b96b34\" (UID: \"155ffa13-2141-4a24-91c5-020772b96b34\") "
Oct 30 23:32:22.391248 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:22.391151    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/155ffa13-2141-4a24-91c5-020772b96b34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155ffa13-2141-4a24-91c5-020772b96b34-maas-api-openshift-ca\") pod \"155ffa13-2141-4a24-91c5-020772b96b34\" (UID: \"155ffa13-2141-4a24-91c5-020772b96b34\") "
Oct 30 23:32:22.391315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:22.391302    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/155ffa13-2141-4a24-91c5-020772b96b34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155ffa13-2141-4a24-91c5-020772b96b34-maas-api-xks-ca\") pod \"155ffa13-2141-4a24-91c5-020772b96b34\" (UID: \"155ffa13-2141-4a24-91c5-020772b96b34\") "
Oct 30 23:32:22.391532 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:22.391503    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/155ffa13-2141-4a24-91c5-020772b96b34-maas-api-openshift-ca" pod "155ffa13-2141-4a24-91c5-020772b96b34" (UID: "155ffa13-2141-4a24-91c5-020772b96b34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:32:22.392695 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:22.392665    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/155ffa13-2141-4a24-91c5-020772b96b34-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:32:22.393464 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:22.393439    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155ffa13-2141-4a24-91c5-020772b96b34-kube-api-access-n2r88" pod "155ffa13-2141-4a24-91c5-020772b96b34" (UID: "155ffa13-2141-4a24-91c5-020772b96b34"). InnerVolumeSpecName "kube-api-access-n2r88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:32:22.393660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:22.393636    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/155ffa13-2141-4a24-91c5-020772b96b34-maas-api-xks-ca" pod "155ffa13-2141-4a24-91c5-020772b96b34" (UID: "155ffa13-2141-4a24-91c5-020772b96b34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:32:22.494429 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:22.494384    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2r88\" (UniqueName: \"kubernetes.io/projected/155ffa13-2141-4a24-91c5-020772b96b34-kube-api-access-n2r88\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:32:22.494429 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:22.494420    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/155ffa13-2141-4a24-91c5-020772b96b34-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:32:23.151955 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:23.151925    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh"
Oct 30 23:32:23.151955 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:23.151945    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh" event={"ID":"155ffa13-2141-4a24-91c5-020772b96b34","Type":"ContainerDied","Data":"358a99a78ff3ef7c7e3fa37487168ee66c0e52857b4ad6a25c47c9e283ff116c"}
Oct 30 23:32:23.152515 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:23.152003    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="358a99a78ff3ef7c7e3fa37487168ee66c0e52857b4ad6a25c47c9e283ff116c"
Oct 30 23:32:23.328546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:23.328519    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc"
Oct 30 23:32:23.407368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:23.407305    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02e3dd03-03e7-42cb-a6b6-3680f039b67b-kube-api-access-4rqlg\" (UniqueName: \"kubernetes.io/projected/02e3dd03-03e7-42cb-a6b6-3680f039b67b-kube-api-access-4rqlg\") pod \"02e3dd03-03e7-42cb-a6b6-3680f039b67b\" (UID: \"02e3dd03-03e7-42cb-a6b6-3680f039b67b\") "
Oct 30 23:32:23.407512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:23.407456    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02e3dd03-03e7-42cb-a6b6-3680f039b67b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e3dd03-03e7-42cb-a6b6-3680f039b67b-maas-api-xks-ca\") pod \"02e3dd03-03e7-42cb-a6b6-3680f039b67b\" (UID: \"02e3dd03-03e7-42cb-a6b6-3680f039b67b\") "
Oct 30 23:32:23.407512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:23.407508    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02e3dd03-03e7-42cb-a6b6-3680f039b67b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e3dd03-03e7-42cb-a6b6-3680f039b67b-maas-api-openshift-ca\") pod \"02e3dd03-03e7-42cb-a6b6-3680f039b67b\" (UID: \"02e3dd03-03e7-42cb-a6b6-3680f039b67b\") "
Oct 30 23:32:23.408020 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:23.407956    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e3dd03-03e7-42cb-a6b6-3680f039b67b-maas-api-openshift-ca" pod "02e3dd03-03e7-42cb-a6b6-3680f039b67b" (UID: "02e3dd03-03e7-42cb-a6b6-3680f039b67b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:32:23.408510 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:23.408488    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02e3dd03-03e7-42cb-a6b6-3680f039b67b-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:32:23.409732 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:23.409704    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02e3dd03-03e7-42cb-a6b6-3680f039b67b-maas-api-xks-ca" pod "02e3dd03-03e7-42cb-a6b6-3680f039b67b" (UID: "02e3dd03-03e7-42cb-a6b6-3680f039b67b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:32:23.409825 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:23.409773    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e3dd03-03e7-42cb-a6b6-3680f039b67b-kube-api-access-4rqlg" pod "02e3dd03-03e7-42cb-a6b6-3680f039b67b" (UID: "02e3dd03-03e7-42cb-a6b6-3680f039b67b"). InnerVolumeSpecName "kube-api-access-4rqlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:32:23.510452 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:23.510413    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rqlg\" (UniqueName: \"kubernetes.io/projected/02e3dd03-03e7-42cb-a6b6-3680f039b67b-kube-api-access-4rqlg\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:32:23.510452 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:23.510448    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02e3dd03-03e7-42cb-a6b6-3680f039b67b-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:32:24.162548 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:24.162510    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc" event={"ID":"02e3dd03-03e7-42cb-a6b6-3680f039b67b","Type":"ContainerDied","Data":"7af67d73be96d71396bf01e81158c909ef189574a78074c812d21464292dd4f5"}
Oct 30 23:32:24.162548 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:24.162539    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc"
Oct 30 23:32:24.162981 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:24.162567    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7af67d73be96d71396bf01e81158c909ef189574a78074c812d21464292dd4f5"
Oct 30 23:32:25.171722 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:25.171653    5078 generic.go:356] "Generic (PLEG): container finished" podID="c562aeb1-976f-4b17-b5b0-db336a33f67f" containerID="4109cab6a90085e24af88132916d95de0d4b1dbac88999a104ab211c3ef12601" exitCode=0
Oct 30 23:32:25.172146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:25.171728    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9" event={"ID":"c562aeb1-976f-4b17-b5b0-db336a33f67f","Type":"ContainerDied","Data":"4109cab6a90085e24af88132916d95de0d4b1dbac88999a104ab211c3ef12601"}
Oct 30 23:32:25.172146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:25.172101    5078 scope.go:122] "RemoveContainer" containerID="4109cab6a90085e24af88132916d95de0d4b1dbac88999a104ab211c3ef12601"
Oct 30 23:32:25.173736 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:25.173706    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:32:25.173810 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:25.173794    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:32:25.173876 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:25.173862    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:32:26.238622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:26.233295    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9" event={"ID":"c562aeb1-976f-4b17-b5b0-db336a33f67f","Type":"ContainerStarted","Data":"21cbe81217a410639e8d43b145ceb94887312bf8e34cd2d238bab3e9cfb8976f"}
Oct 30 23:32:26.879822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:26.874709    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:32:26.879822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:26.874844    5078 patch_prober.go:29] interesting pod/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 23:32:26.879822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:26.874902    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9" podUID="c562aeb1-976f-4b17-b5b0-db336a33f67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 23:32:27.874618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:27.873940    5078 patch_prober.go:29] interesting pod/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 23:32:27.874618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:27.873997    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9" podUID="c562aeb1-976f-4b17-b5b0-db336a33f67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 23:32:28.037297 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:28.037263    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l"]
Oct 30 23:32:28.048397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:28.048368    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-49d07c83-f4q9l"]
Oct 30 23:32:28.538383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:28.538345    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d481549e-d6a4-44a7-ac85-98c466dbab14" path="/var/lib/kubelet/pods/d481549e-d6a4-44a7-ac85-98c466dbab14/volumes"
Oct 30 23:32:28.873618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:28.868929    5078 patch_prober.go:29] interesting pod/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 23:32:28.873618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:28.868992    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9" podUID="c562aeb1-976f-4b17-b5b0-db336a33f67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 23:32:29.786620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:29.783300    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"]
Oct 30 23:32:29.786620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:29.784225    5078 cpu_manager.go:414] "containerMap: removing container" podUID="02e3dd03-03e7-42cb-a6b6-3680f039b67b" containerName="revoke-keys"
Oct 30 23:32:29.786620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:29.784244    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02e3dd03-03e7-42cb-a6b6-3680f039b67b" containerName="revoke-keys"
Oct 30 23:32:29.786620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:29.784278    5078 cpu_manager.go:414] "containerMap: removing container" podUID="155ffa13-2141-4a24-91c5-020772b96b34" containerName="revoke-keys"
Oct 30 23:32:29.786620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:29.784284    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="155ffa13-2141-4a24-91c5-020772b96b34" containerName="revoke-keys"
Oct 30 23:32:29.786620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:29.784444    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="02e3dd03-03e7-42cb-a6b6-3680f039b67b" containerName="revoke-keys"
Oct 30 23:32:29.786620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:29.784459    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="155ffa13-2141-4a24-91c5-020772b96b34" containerName="revoke-keys"
Oct 30 23:32:29.911151 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:29.911114    5078 patch_prober.go:29] interesting pod/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 23:32:29.913875 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:29.913837    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9" podUID="c562aeb1-976f-4b17-b5b0-db336a33f67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 23:32:29.929689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:29.928108    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.078751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.063089    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p"]
Oct 30 23:32:30.134792 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.134695    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-20f33adb-gw-openshift-default-dockercfg-tbsbt\""
Oct 30 23:32:30.201293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.201200    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"]
Oct 30 23:32:30.210365 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.210317    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv"]
Oct 30 23:32:30.232765 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.232734    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-workload-certs\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.234217 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.234188    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-envoy\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.234923 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.234901    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-credential-socket\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.235141 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.235124    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-token\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.235367 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.235351    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istiod-ca-cert\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.235514 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.235500    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nflk\" (UniqueName: \"kubernetes.io/projected/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-kube-api-access-7nflk\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.235832 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.235814    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-workload-socket\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.236055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.236038    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-data\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.236293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.236277    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-podinfo\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.453701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.450219    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-envoy\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.453701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.450284    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-credential-socket\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.453701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.450315    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-token\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.453701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.450409    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istiod-ca-cert\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.453701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.450464    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nflk\" (UniqueName: \"kubernetes.io/projected/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-kube-api-access-7nflk\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.453701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.450509    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-workload-socket\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.453701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.450557    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-data\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.453701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.450669    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-podinfo\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.453701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.450807    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-workload-certs\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.453701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.452041    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-data\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.453701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.452632    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-workload-socket\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.453701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.452651    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istiod-ca-cert\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.453701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.452877    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-workload-certs\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.453701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.453317    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-credential-socket\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.457963 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.457937    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-envoy\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.530702 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.527141    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-podinfo\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.585698 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.569277    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-token\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.586304 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.586261    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nflk\" (UniqueName: \"kubernetes.io/projected/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-kube-api-access-7nflk\") pod \"e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") " pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.834111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.834077    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:30.882882 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.882850    5078 patch_prober.go:29] interesting pod/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 23:32:30.883075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:30.882911    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9" podUID="c562aeb1-976f-4b17-b5b0-db336a33f67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 23:32:31.053378 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:31.053344    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5"]
Oct 30 23:32:31.084955 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:31.084868    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d06b7cea-tclh5"]
Oct 30 23:32:31.291460 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:32:31.291430    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b655ccf_f1c3_43e8_82b7_259f9a9cf0e7.slice/crio-0bc047a0e77981e3f9f3cf794591520e908d873e325370063df526f5055ecda7 WatchSource:0}: Error finding container 0bc047a0e77981e3f9f3cf794591520e908d873e325370063df526f5055ecda7: Status 404 returned error can't find the container with id 0bc047a0e77981e3f9f3cf794591520e908d873e325370063df526f5055ecda7
Oct 30 23:32:31.293007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:31.292982    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"]
Oct 30 23:32:31.294178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:31.294146    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:32:31.294236 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:31.294207    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:32:31.294274 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:31.294236    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:32:31.756313 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:31.756190    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm" event={"ID":"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7","Type":"ContainerStarted","Data":"507a1dc6ac43513a08207acb61a5b7ac23838e3b2ffc993d75f01002fba6bce4"}
Oct 30 23:32:31.756576 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:31.756553    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm" event={"ID":"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7","Type":"ContainerStarted","Data":"0bc047a0e77981e3f9f3cf794591520e908d873e325370063df526f5055ecda7"}
Oct 30 23:32:31.839621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:31.837503    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:31.870392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:31.869969    5078 patch_prober.go:29] interesting pod/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Oct 30 23:32:31.870392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:31.870028    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9" podUID="c562aeb1-976f-4b17-b5b0-db336a33f67f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Oct 30 23:32:32.408345 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:32.366208    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c573c188-ec6c-4465-98fd-edb374cd13f7" path="/var/lib/kubelet/pods/c573c188-ec6c-4465-98fd-edb374cd13f7/volumes"
Oct 30 23:32:32.840892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:32.840866    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:32.869756 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:32.869729    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:32:32.870047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:32.870025    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:32:32.870659 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:32.870644    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:32:33.011404 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:33.011354    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm" podStartSLOduration=4.011335256 podStartE2EDuration="4.011335256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:32:31.92513483 +0000 UTC m=+1537.813041196" watchObservedRunningTime="2026-10-30 23:32:33.011335256 +0000 UTC m=+1538.899241609"
Oct 30 23:32:33.777470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:33.777436    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:33.778263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:33.778244    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:32:35.078315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.078280    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p" podUID="382a4000-3d58-4e53-b506-19df51a78e42" containerName="payload-processing" containerID="cri-o://be6af15d29595c8b270e0ccc5bfa99502f95049e554b552a06a661923cef171e" gracePeriod=25
Oct 30 23:32:35.211908 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.211868    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv" podUID="e231029b-bdb1-4433-ba72-0c5471c554a6" containerName="payload-pre-processing" containerID="cri-o://2901a5c88b9b15cdf6d373712bd13fd221f6cb3b0fe93fa3e6e1dad3028849e0" gracePeriod=25
Oct 30 23:32:35.372773 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.372443    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p"
Oct 30 23:32:35.407323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.407294    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/382a4000-3d58-4e53-b506-19df51a78e42-kube-api-access-qqdfz\" (UniqueName: \"kubernetes.io/projected/382a4000-3d58-4e53-b506-19df51a78e42-kube-api-access-qqdfz\") pod \"382a4000-3d58-4e53-b506-19df51a78e42\" (UID: \"382a4000-3d58-4e53-b506-19df51a78e42\") "
Oct 30 23:32:35.409849 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.409806    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/382a4000-3d58-4e53-b506-19df51a78e42-kube-api-access-qqdfz" pod "382a4000-3d58-4e53-b506-19df51a78e42" (UID: "382a4000-3d58-4e53-b506-19df51a78e42"). InnerVolumeSpecName "kube-api-access-qqdfz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:32:35.502128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.502104    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv"
Oct 30 23:32:35.508667 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.508644    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/382a4000-3d58-4e53-b506-19df51a78e42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382a4000-3d58-4e53-b506-19df51a78e42-plugins-config-volume\") pod \"382a4000-3d58-4e53-b506-19df51a78e42\" (UID: \"382a4000-3d58-4e53-b506-19df51a78e42\") "
Oct 30 23:32:35.508818 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.508775    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e231029b-bdb1-4433-ba72-0c5471c554a6-kube-api-access-ztslv\" (UniqueName: \"kubernetes.io/projected/e231029b-bdb1-4433-ba72-0c5471c554a6-kube-api-access-ztslv\") pod \"e231029b-bdb1-4433-ba72-0c5471c554a6\" (UID: \"e231029b-bdb1-4433-ba72-0c5471c554a6\") "
Oct 30 23:32:35.508886 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.508871    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e231029b-bdb1-4433-ba72-0c5471c554a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e231029b-bdb1-4433-ba72-0c5471c554a6-plugins-config-volume\") pod \"e231029b-bdb1-4433-ba72-0c5471c554a6\" (UID: \"e231029b-bdb1-4433-ba72-0c5471c554a6\") "
Oct 30 23:32:35.509085 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.509059    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/382a4000-3d58-4e53-b506-19df51a78e42-plugins-config-volume" pod "382a4000-3d58-4e53-b506-19df51a78e42" (UID: "382a4000-3d58-4e53-b506-19df51a78e42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:32:35.509370 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.509348    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e231029b-bdb1-4433-ba72-0c5471c554a6-plugins-config-volume" pod "e231029b-bdb1-4433-ba72-0c5471c554a6" (UID: "e231029b-bdb1-4433-ba72-0c5471c554a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:32:35.509611 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.509579    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqdfz\" (UniqueName: \"kubernetes.io/projected/382a4000-3d58-4e53-b506-19df51a78e42-kube-api-access-qqdfz\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:32:35.509690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.509623    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e231029b-bdb1-4433-ba72-0c5471c554a6-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:32:35.509690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.509639    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/382a4000-3d58-4e53-b506-19df51a78e42-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:32:35.511639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.511612    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e231029b-bdb1-4433-ba72-0c5471c554a6-kube-api-access-ztslv" pod "e231029b-bdb1-4433-ba72-0c5471c554a6" (UID: "e231029b-bdb1-4433-ba72-0c5471c554a6"). InnerVolumeSpecName "kube-api-access-ztslv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:32:35.611100 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.611031    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztslv\" (UniqueName: \"kubernetes.io/projected/e231029b-bdb1-4433-ba72-0c5471c554a6-kube-api-access-ztslv\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:32:35.793035 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.793004    5078 generic.go:356] "Generic (PLEG): container finished" podID="382a4000-3d58-4e53-b506-19df51a78e42" containerID="be6af15d29595c8b270e0ccc5bfa99502f95049e554b552a06a661923cef171e" exitCode=0
Oct 30 23:32:35.793210 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.793053    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p"
Oct 30 23:32:35.793210 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.793098    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p" event={"ID":"382a4000-3d58-4e53-b506-19df51a78e42","Type":"ContainerDied","Data":"be6af15d29595c8b270e0ccc5bfa99502f95049e554b552a06a661923cef171e"}
Oct 30 23:32:35.793210 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.793179    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p" event={"ID":"382a4000-3d58-4e53-b506-19df51a78e42","Type":"ContainerDied","Data":"3d323acfe559f56dca65fba78c2eda4f422d225b55c68737531f4e16cd506c8c"}
Oct 30 23:32:35.793210 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.793206    5078 scope.go:122] "RemoveContainer" containerID="be6af15d29595c8b270e0ccc5bfa99502f95049e554b552a06a661923cef171e"
Oct 30 23:32:35.794942 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.794924    5078 generic.go:356] "Generic (PLEG): container finished" podID="e231029b-bdb1-4433-ba72-0c5471c554a6" containerID="2901a5c88b9b15cdf6d373712bd13fd221f6cb3b0fe93fa3e6e1dad3028849e0" exitCode=0
Oct 30 23:32:35.795041 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.794991    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv" event={"ID":"e231029b-bdb1-4433-ba72-0c5471c554a6","Type":"ContainerDied","Data":"2901a5c88b9b15cdf6d373712bd13fd221f6cb3b0fe93fa3e6e1dad3028849e0"}
Oct 30 23:32:35.795041 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.795032    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv" event={"ID":"e231029b-bdb1-4433-ba72-0c5471c554a6","Type":"ContainerDied","Data":"1cdbedba397662996cd9ace36f8438711e530dc198abe331b33c301123439971"}
Oct 30 23:32:35.795227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.795217    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv"
Oct 30 23:32:35.807034 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.807016    5078 scope.go:122] "RemoveContainer" containerID="be6af15d29595c8b270e0ccc5bfa99502f95049e554b552a06a661923cef171e"
Oct 30 23:32:35.807320 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:32:35.807303    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6af15d29595c8b270e0ccc5bfa99502f95049e554b552a06a661923cef171e\": container with ID starting with be6af15d29595c8b270e0ccc5bfa99502f95049e554b552a06a661923cef171e not found: ID does not exist" containerID="be6af15d29595c8b270e0ccc5bfa99502f95049e554b552a06a661923cef171e"
Oct 30 23:32:35.807380 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.807329    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6af15d29595c8b270e0ccc5bfa99502f95049e554b552a06a661923cef171e"} err="failed to get container status \"be6af15d29595c8b270e0ccc5bfa99502f95049e554b552a06a661923cef171e\": rpc error: code = NotFound desc = could not find container \"be6af15d29595c8b270e0ccc5bfa99502f95049e554b552a06a661923cef171e\": container with ID starting with be6af15d29595c8b270e0ccc5bfa99502f95049e554b552a06a661923cef171e not found: ID does not exist"
Oct 30 23:32:35.807380 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.807345    5078 scope.go:122] "RemoveContainer" containerID="2901a5c88b9b15cdf6d373712bd13fd221f6cb3b0fe93fa3e6e1dad3028849e0"
Oct 30 23:32:35.817073 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.817058    5078 scope.go:122] "RemoveContainer" containerID="2901a5c88b9b15cdf6d373712bd13fd221f6cb3b0fe93fa3e6e1dad3028849e0"
Oct 30 23:32:35.817351 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:32:35.817336    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2901a5c88b9b15cdf6d373712bd13fd221f6cb3b0fe93fa3e6e1dad3028849e0\": container with ID starting with 2901a5c88b9b15cdf6d373712bd13fd221f6cb3b0fe93fa3e6e1dad3028849e0 not found: ID does not exist" containerID="2901a5c88b9b15cdf6d373712bd13fd221f6cb3b0fe93fa3e6e1dad3028849e0"
Oct 30 23:32:35.817392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.817358    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2901a5c88b9b15cdf6d373712bd13fd221f6cb3b0fe93fa3e6e1dad3028849e0"} err="failed to get container status \"2901a5c88b9b15cdf6d373712bd13fd221f6cb3b0fe93fa3e6e1dad3028849e0\": rpc error: code = NotFound desc = could not find container \"2901a5c88b9b15cdf6d373712bd13fd221f6cb3b0fe93fa3e6e1dad3028849e0\": container with ID starting with 2901a5c88b9b15cdf6d373712bd13fd221f6cb3b0fe93fa3e6e1dad3028849e0 not found: ID does not exist"
Oct 30 23:32:35.961135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.961051    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv"]
Oct 30 23:32:35.966679 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:35.966641    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-40c57b52-7bcd859b6c-d5fcv"]
Oct 30 23:32:36.076957 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:36.076928    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p"]
Oct 30 23:32:36.081732 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:36.081711    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-40c57b52-84c6d56b59-s2z5p"]
Oct 30 23:32:36.299569 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:36.299529    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="382a4000-3d58-4e53-b506-19df51a78e42" path="/var/lib/kubelet/pods/382a4000-3d58-4e53-b506-19df51a78e42/volumes"
Oct 30 23:32:36.300074 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:36.300059    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e231029b-bdb1-4433-ba72-0c5471c554a6" path="/var/lib/kubelet/pods/e231029b-bdb1-4433-ba72-0c5471c554a6/volumes"
Oct 30 23:32:37.402474 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.402432    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d"]
Oct 30 23:32:37.404331 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.404302    5078 cpu_manager.go:414] "containerMap: removing container" podUID="382a4000-3d58-4e53-b506-19df51a78e42" containerName="payload-processing"
Oct 30 23:32:37.404493 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.404479    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="382a4000-3d58-4e53-b506-19df51a78e42" containerName="payload-processing"
Oct 30 23:32:37.404590 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.404579    5078 cpu_manager.go:414] "containerMap: removing container" podUID="e231029b-bdb1-4433-ba72-0c5471c554a6" containerName="payload-pre-processing"
Oct 30 23:32:37.404713 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.404699    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e231029b-bdb1-4433-ba72-0c5471c554a6" containerName="payload-pre-processing"
Oct 30 23:32:37.405007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.404991    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="382a4000-3d58-4e53-b506-19df51a78e42" containerName="payload-processing"
Oct 30 23:32:37.405115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.405104    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="e231029b-bdb1-4433-ba72-0c5471c554a6" containerName="payload-pre-processing"
Oct 30 23:32:37.501631 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.480497    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d"
Oct 30 23:32:37.581190 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.581141    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-20f33adb\""
Oct 30 23:32:37.754342 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.581716    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-20f33adb-dockercfg-wqbbj\""
Oct 30 23:32:37.823817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.823783    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4386f5ab-5a83-4be1-af56-255e4ee9b5da-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d\" (UID: \"4386f5ab-5a83-4be1-af56-255e4ee9b5da\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d"
Oct 30 23:32:37.824358 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.824329    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwvx8\" (UniqueName: \"kubernetes.io/projected/4386f5ab-5a83-4be1-af56-255e4ee9b5da-kube-api-access-jwvx8\") pod \"payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d\" (UID: \"4386f5ab-5a83-4be1-af56-255e4ee9b5da\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d"
Oct 30 23:32:37.840179 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.840154    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d"]
Oct 30 23:32:37.887748 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.887716    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr"]
Oct 30 23:32:37.893518 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.893486    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr"
Oct 30 23:32:37.895284 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.895258    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr"]
Oct 30 23:32:37.950757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.950720    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr\" (UID: \"1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b\") " pod="openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr"
Oct 30 23:32:37.951286 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.951259    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4386f5ab-5a83-4be1-af56-255e4ee9b5da-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d\" (UID: \"4386f5ab-5a83-4be1-af56-255e4ee9b5da\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d"
Oct 30 23:32:37.951530 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.951513    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gq5j\" (UniqueName: \"kubernetes.io/projected/1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b-kube-api-access-7gq5j\") pod \"payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr\" (UID: \"1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b\") " pod="openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr"
Oct 30 23:32:37.951766 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.951749    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwvx8\" (UniqueName: \"kubernetes.io/projected/4386f5ab-5a83-4be1-af56-255e4ee9b5da-kube-api-access-jwvx8\") pod \"payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d\" (UID: \"4386f5ab-5a83-4be1-af56-255e4ee9b5da\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d"
Oct 30 23:32:37.990673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:37.990639    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4386f5ab-5a83-4be1-af56-255e4ee9b5da-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d\" (UID: \"4386f5ab-5a83-4be1-af56-255e4ee9b5da\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d"
Oct 30 23:32:38.076662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:38.076542    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwvx8\" (UniqueName: \"kubernetes.io/projected/4386f5ab-5a83-4be1-af56-255e4ee9b5da-kube-api-access-jwvx8\") pod \"payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d\" (UID: \"4386f5ab-5a83-4be1-af56-255e4ee9b5da\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d"
Oct 30 23:32:38.080628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:38.077570    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gq5j\" (UniqueName: \"kubernetes.io/projected/1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b-kube-api-access-7gq5j\") pod \"payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr\" (UID: \"1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b\") " pod="openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr"
Oct 30 23:32:38.080628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:38.077736    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr\" (UID: \"1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b\") " pod="openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr"
Oct 30 23:32:38.098338 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:38.090198    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b-plugins-config-volume\") pod \"payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr\" (UID: \"1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b\") " pod="openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr"
Oct 30 23:32:38.191161 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:38.183325    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gq5j\" (UniqueName: \"kubernetes.io/projected/1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b-kube-api-access-7gq5j\") pod \"payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr\" (UID: \"1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b\") " pod="openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr"
Oct 30 23:32:38.191161 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:38.183635    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d"
Oct 30 23:32:38.229624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:38.221160    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr"
Oct 30 23:32:38.998768 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:38.996170    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d"]
Oct 30 23:32:39.121057 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:32:39.121023    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf17ea6_8cc2_4a4b_9e1f_ceeacc389f9b.slice/crio-9833b91d6ad1d2cb8e8357575405a3ea904a3080ba81523dbeab3d7410adae24 WatchSource:0}: Error finding container 9833b91d6ad1d2cb8e8357575405a3ea904a3080ba81523dbeab3d7410adae24: Status 404 returned error can't find the container with id 9833b91d6ad1d2cb8e8357575405a3ea904a3080ba81523dbeab3d7410adae24
Oct 30 23:32:39.122291 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:39.122270    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr"]
Oct 30 23:32:40.046637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:40.044497    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5"]
Oct 30 23:32:40.088993 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:40.088953    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d" event={"ID":"4386f5ab-5a83-4be1-af56-255e4ee9b5da","Type":"ContainerStarted","Data":"c69ed29a03136a5424d5394ef0bce65767931efec76d4685dd8b7d8ae8e1e7a9"}
Oct 30 23:32:40.089327 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:40.089267    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d" event={"ID":"4386f5ab-5a83-4be1-af56-255e4ee9b5da","Type":"ContainerStarted","Data":"c776ab6457270c83b111e6a08baa1413e0a5a141d8bd4ff484430740e7fe0ad3"}
Oct 30 23:32:40.090523 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:40.090500    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d"
Oct 30 23:32:40.094026 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:40.093998    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr" event={"ID":"1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b","Type":"ContainerStarted","Data":"8a8d620328edf96328187e5d246a3e0658b99efa8aa634488e84de68d870ba2b"}
Oct 30 23:32:40.094199 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:40.094182    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr" event={"ID":"1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b","Type":"ContainerStarted","Data":"9833b91d6ad1d2cb8e8357575405a3ea904a3080ba81523dbeab3d7410adae24"}
Oct 30 23:32:40.095189 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:40.095167    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr"
Oct 30 23:32:40.627155 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:40.623374    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d" podStartSLOduration=3.623351568 podStartE2EDuration="3.623351568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:32:40.213714562 +0000 UTC m=+1546.101620927" watchObservedRunningTime="2026-10-30 23:32:40.623351568 +0000 UTC m=+1546.511257925"
Oct 30 23:32:40.663621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:40.660404    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr" podStartSLOduration=3.660384163 podStartE2EDuration="3.660384163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:32:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:32:40.339067358 +0000 UTC m=+1546.226973712" watchObservedRunningTime="2026-10-30 23:32:40.660384163 +0000 UTC m=+1546.548290517"
Oct 30 23:32:40.695633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:40.695003    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-01c440ca-9twp5"]
Oct 30 23:32:41.048156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:41.048125    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw"]
Oct 30 23:32:41.065643 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:41.065616    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9bb1a5d5-xtscw"]
Oct 30 23:32:41.346187 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:41.346108    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95"]
Oct 30 23:32:41.351752 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:41.351724    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2b6c82a0-jmv95"]
Oct 30 23:32:41.984775 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:41.984732    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"]
Oct 30 23:32:42.229989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.229957    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"]
Oct 30 23:32:42.265903 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.230752    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.397871 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.393725    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-196ff7-openshift-default-dockercfg-6g94v\""
Oct 30 23:32:42.435740 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.435702    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402" path="/var/lib/kubelet/pods/2bbd7ee7-d0e1-4a35-bd59-7bd64b5a9402/volumes"
Oct 30 23:32:42.436947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.436923    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea74946-96bd-42de-8c06-ea230836fdd3" path="/var/lib/kubelet/pods/8ea74946-96bd-42de-8c06-ea230836fdd3/volumes"
Oct 30 23:32:42.437993 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.437972    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0b479e9-979a-4251-9278-f478dec4f82f" path="/var/lib/kubelet/pods/f0b479e9-979a-4251-9278-f478dec4f82f/volumes"
Oct 30 23:32:42.540725 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.540691    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qbqg\" (UniqueName: \"kubernetes.io/projected/fc173733-80d8-4793-b1a1-723844b99673-kube-api-access-7qbqg\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.540908 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.540871    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc173733-80d8-4793-b1a1-723844b99673-istiod-ca-cert\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.540972 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.540927    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc173733-80d8-4793-b1a1-723844b99673-istio-podinfo\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.541037 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.541021    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-credential-socket\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.541098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.541056    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-istio-data\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.541240 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.541211    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc173733-80d8-4793-b1a1-723844b99673-istio-token\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.541304 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.541281    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-workload-socket\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.541493 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.541416    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-istio-envoy\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.541493 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.541458    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-workload-certs\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.642367 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.642334    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-workload-socket\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.642559 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.642418    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-istio-envoy\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.642559 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.642454    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-workload-certs\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.642559 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.642534    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qbqg\" (UniqueName: \"kubernetes.io/projected/fc173733-80d8-4793-b1a1-723844b99673-kube-api-access-7qbqg\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.642785 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.642691    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc173733-80d8-4793-b1a1-723844b99673-istiod-ca-cert\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.642785 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.642747    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc173733-80d8-4793-b1a1-723844b99673-istio-podinfo\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.642892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.642816    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-credential-socket\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.642892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.642838    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-workload-socket\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.642892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.642846    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-istio-data\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.643056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.642923    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc173733-80d8-4793-b1a1-723844b99673-istio-token\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.643109 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.643085    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-istio-data\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.643395 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.643366    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-workload-certs\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.643539 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.643516    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc173733-80d8-4793-b1a1-723844b99673-istiod-ca-cert\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.643664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.643632    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-credential-socket\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.645135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.645110    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-istio-envoy\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.645519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.645496    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc173733-80d8-4793-b1a1-723844b99673-istio-podinfo\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.701312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.701277    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc173733-80d8-4793-b1a1-723844b99673-istio-token\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.701477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.701391    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qbqg\" (UniqueName: \"kubernetes.io/projected/fc173733-80d8-4793-b1a1-723844b99673-kube-api-access-7qbqg\") pod \"e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") " pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:42.940773 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:42.940678    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:43.750851 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:43.750766    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"]
Oct 30 23:32:43.755148 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:32:43.755119    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc173733_80d8_4793_b1a1_723844b99673.slice/crio-e052506ba69888d254476d1df6b7215028d490b3a9deb2bcf8e97fad0c3f9835 WatchSource:0}: Error finding container e052506ba69888d254476d1df6b7215028d490b3a9deb2bcf8e97fad0c3f9835: Status 404 returned error can't find the container with id e052506ba69888d254476d1df6b7215028d490b3a9deb2bcf8e97fad0c3f9835
Oct 30 23:32:43.758158 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:43.758119    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:32:43.758260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:43.758207    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:32:43.758260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:43.758248    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:32:44.313655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:44.313579    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r" event={"ID":"fc173733-80d8-4793-b1a1-723844b99673","Type":"ContainerStarted","Data":"a9a1da734944bbd0a55c88375283d87a7fc4634a4df94df2bd704ab535c62b27"}
Oct 30 23:32:44.313848 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:44.313671    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r" event={"ID":"fc173733-80d8-4793-b1a1-723844b99673","Type":"ContainerStarted","Data":"e052506ba69888d254476d1df6b7215028d490b3a9deb2bcf8e97fad0c3f9835"}
Oct 30 23:32:44.486960 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:44.486911    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r" podStartSLOduration=3.486896022 podStartE2EDuration="3.486896022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:32:44.483570321 +0000 UTC m=+1550.371476676" watchObservedRunningTime="2026-10-30 23:32:44.486896022 +0000 UTC m=+1550.374802375"
Oct 30 23:32:44.945279 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:44.945251    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:44.950103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:44.950082    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:45.322215 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:45.322193    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:45.323073 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:45.323056    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:32:47.593570 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:32:47.593531    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:32:51.054711 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:51.051179    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw"]
Oct 30 23:32:51.074646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:51.061886    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw"
Oct 30 23:32:51.074646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:51.072803    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcklw\" (UniqueName: \"kubernetes.io/projected/72d8a560-396f-486c-b602-5496dfa56ada-kube-api-access-bcklw\") pod \"payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw\" (UID: \"72d8a560-396f-486c-b602-5496dfa56ada\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw"
Oct 30 23:32:51.074646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:51.073135    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d8a560-396f-486c-b602-5496dfa56ada-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw\" (UID: \"72d8a560-396f-486c-b602-5496dfa56ada\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw"
Oct 30 23:32:51.141865 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:51.141128    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-196ff7-dockercfg-nzzdg\""
Oct 30 23:32:51.141865 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:51.141823    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-196ff7\""
Oct 30 23:32:51.148418 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:51.148374    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw"]
Oct 30 23:32:51.175314 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:51.175279    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d8a560-396f-486c-b602-5496dfa56ada-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw\" (UID: \"72d8a560-396f-486c-b602-5496dfa56ada\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw"
Oct 30 23:32:51.176523 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:51.176494    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d8a560-396f-486c-b602-5496dfa56ada-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw\" (UID: \"72d8a560-396f-486c-b602-5496dfa56ada\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw"
Oct 30 23:32:51.177076 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:51.176731    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcklw\" (UniqueName: \"kubernetes.io/projected/72d8a560-396f-486c-b602-5496dfa56ada-kube-api-access-bcklw\") pod \"payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw\" (UID: \"72d8a560-396f-486c-b602-5496dfa56ada\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw"
Oct 30 23:32:51.530677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:51.483309    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcklw\" (UniqueName: \"kubernetes.io/projected/72d8a560-396f-486c-b602-5496dfa56ada-kube-api-access-bcklw\") pod \"payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw\" (UID: \"72d8a560-396f-486c-b602-5496dfa56ada\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw"
Oct 30 23:32:51.716940 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:51.716902    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw"
Oct 30 23:32:52.151521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:52.151489    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw"]
Oct 30 23:32:52.265514 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:52.265490    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr"
Oct 30 23:32:52.265686 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:52.265636    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d"
Oct 30 23:32:52.484477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:52.484388    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw" event={"ID":"72d8a560-396f-486c-b602-5496dfa56ada","Type":"ContainerStarted","Data":"5fc787b400adbae5cc4ab6fb2dfc46647b00e9fddc72ae57ffb5fdbfa2e7ae0f"}
Oct 30 23:32:52.484671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:52.484480    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw" event={"ID":"72d8a560-396f-486c-b602-5496dfa56ada","Type":"ContainerStarted","Data":"e4509eb3bcacafbc0c06cad08bd62245705e4a766a40634cc67310b3dd66f9da"}
Oct 30 23:32:52.484671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:52.484504    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw"
Oct 30 23:32:52.654165 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:52.654099    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw" podStartSLOduration=2.654074546 podStartE2EDuration="2.654074546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:32:52.622048257 +0000 UTC m=+1558.509954610" watchObservedRunningTime="2026-10-30 23:32:52.654074546 +0000 UTC m=+1558.541980900"
Oct 30 23:32:55.207322 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:55.207242    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw"]
Oct 30 23:32:55.295939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:55.295896    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d523e83-m4ljw"]
Oct 30 23:32:56.344619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:56.344028    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52318041-8b2f-4a12-9ba4-da9e1a0bb135" path="/var/lib/kubelet/pods/52318041-8b2f-4a12-9ba4-da9e1a0bb135/volumes"
Oct 30 23:32:57.971620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:32:57.970659    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:33:00.632895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:00.632859    5078 scope.go:122] "RemoveContainer" containerID="ba0434a839ec161f63ffc457008cb6f6ce64737dbee375bd777f76a447b14047"
Oct 30 23:33:00.643731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:00.643706    5078 scope.go:122] "RemoveContainer" containerID="b914aa5183530f6f654abaa599ba97537e490aa36f81db25bc2073a669faffec"
Oct 30 23:33:03.495494 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:03.495465    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw"
Oct 30 23:33:08.955454 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:08.955424    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh"]
Oct 30 23:33:09.004319 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:09.004291    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-40c57b52-27mkh"]
Oct 30 23:33:09.666620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:09.662690    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"]
Oct 30 23:33:09.666620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:09.662952    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9" podUID="c562aeb1-976f-4b17-b5b0-db336a33f67f" containerName="istio-proxy" containerID="cri-o://21cbe81217a410639e8d43b145ceb94887312bf8e34cd2d238bab3e9cfb8976f" gracePeriod=30
Oct 30 23:33:10.299676 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:10.299637    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="155ffa13-2141-4a24-91c5-020772b96b34" path="/var/lib/kubelet/pods/155ffa13-2141-4a24-91c5-020772b96b34/volumes"
Oct 30 23:33:13.047716 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:13.047684    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql"]
Oct 30 23:33:13.097140 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:13.097112    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34ef737b-fktql"]
Oct 30 23:33:14.049136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:14.049107    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl"]
Oct 30 23:33:14.097249 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:14.097217    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-356b0f8d-qgmpl"]
Oct 30 23:33:14.299587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:14.299502    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fabb604-c8a4-4e4e-a1ee-690b494fda35" path="/var/lib/kubelet/pods/0fabb604-c8a4-4e4e-a1ee-690b494fda35/volumes"
Oct 30 23:33:14.300073 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:14.300057    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de16dc9d-9d25-4439-9b64-3f40d5e2bc70" path="/var/lib/kubelet/pods/de16dc9d-9d25-4439-9b64-3f40d5e2bc70/volumes"
Oct 30 23:33:15.057869 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.057832    5078 generic.go:356] "Generic (PLEG): container finished" podID="c562aeb1-976f-4b17-b5b0-db336a33f67f" containerID="21cbe81217a410639e8d43b145ceb94887312bf8e34cd2d238bab3e9cfb8976f" exitCode=0
Oct 30 23:33:15.058344 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.057880    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9" event={"ID":"c562aeb1-976f-4b17-b5b0-db336a33f67f","Type":"ContainerDied","Data":"21cbe81217a410639e8d43b145ceb94887312bf8e34cd2d238bab3e9cfb8976f"}
Oct 30 23:33:15.058344 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.057977    5078 scope.go:122] "RemoveContainer" containerID="4109cab6a90085e24af88132916d95de0d4b1dbac88999a104ab211c3ef12601"
Oct 30 23:33:15.181379 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.181316    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:33:15.227460 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.227429    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-envoy\") pod \"c562aeb1-976f-4b17-b5b0-db336a33f67f\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") "
Oct 30 23:33:15.227657 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.227531    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-credential-socket\") pod \"c562aeb1-976f-4b17-b5b0-db336a33f67f\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") "
Oct 30 23:33:15.227896 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.227860    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-credential-socket" pod "c562aeb1-976f-4b17-b5b0-db336a33f67f" (UID: "c562aeb1-976f-4b17-b5b0-db336a33f67f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:33:15.228009 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.227918    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-podinfo\") pod \"c562aeb1-976f-4b17-b5b0-db336a33f67f\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") "
Oct 30 23:33:15.228424 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.228395    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-workload-certs" pod "c562aeb1-976f-4b17-b5b0-db336a33f67f" (UID: "c562aeb1-976f-4b17-b5b0-db336a33f67f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:33:15.229925 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.229903    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-workload-certs\") pod \"c562aeb1-976f-4b17-b5b0-db336a33f67f\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") "
Oct 30 23:33:15.230011 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.229998    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-workload-socket\") pod \"c562aeb1-976f-4b17-b5b0-db336a33f67f\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") "
Oct 30 23:33:15.230108 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.230031    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c562aeb1-976f-4b17-b5b0-db336a33f67f-kube-api-access-gqx6s\" (UniqueName: \"kubernetes.io/projected/c562aeb1-976f-4b17-b5b0-db336a33f67f-kube-api-access-gqx6s\") pod \"c562aeb1-976f-4b17-b5b0-db336a33f67f\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") "
Oct 30 23:33:15.230198 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.230181    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-data\") pod \"c562aeb1-976f-4b17-b5b0-db336a33f67f\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") "
Oct 30 23:33:15.230583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.230383    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-workload-socket" pod "c562aeb1-976f-4b17-b5b0-db336a33f67f" (UID: "c562aeb1-976f-4b17-b5b0-db336a33f67f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:33:15.230583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.230415    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-envoy" pod "c562aeb1-976f-4b17-b5b0-db336a33f67f" (UID: "c562aeb1-976f-4b17-b5b0-db336a33f67f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:33:15.230583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.230561    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-data" pod "c562aeb1-976f-4b17-b5b0-db336a33f67f" (UID: "c562aeb1-976f-4b17-b5b0-db336a33f67f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:33:15.230816 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.230612    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-token\" (UniqueName: \"kubernetes.io/projected/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-token\") pod \"c562aeb1-976f-4b17-b5b0-db336a33f67f\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") "
Oct 30 23:33:15.231058 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.231038    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c562aeb1-976f-4b17-b5b0-db336a33f67f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c562aeb1-976f-4b17-b5b0-db336a33f67f-istiod-ca-cert\") pod \"c562aeb1-976f-4b17-b5b0-db336a33f67f\" (UID: \"c562aeb1-976f-4b17-b5b0-db336a33f67f\") "
Oct 30 23:33:15.231406 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.231129    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-podinfo" pod "c562aeb1-976f-4b17-b5b0-db336a33f67f" (UID: "c562aeb1-976f-4b17-b5b0-db336a33f67f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:33:15.231480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.231447    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c562aeb1-976f-4b17-b5b0-db336a33f67f-istiod-ca-cert" pod "c562aeb1-976f-4b17-b5b0-db336a33f67f" (UID: "c562aeb1-976f-4b17-b5b0-db336a33f67f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:33:15.232370 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.232340    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:33:15.232370 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.232367    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:33:15.232546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.232383    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:33:15.232546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.232396    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:33:15.232546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.232410    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:33:15.232546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.232423    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:33:15.232546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.232436    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c562aeb1-976f-4b17-b5b0-db336a33f67f-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:33:15.233091 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.232812    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-token" pod "c562aeb1-976f-4b17-b5b0-db336a33f67f" (UID: "c562aeb1-976f-4b17-b5b0-db336a33f67f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:33:15.234667 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.234641    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c562aeb1-976f-4b17-b5b0-db336a33f67f-kube-api-access-gqx6s" pod "c562aeb1-976f-4b17-b5b0-db336a33f67f" (UID: "c562aeb1-976f-4b17-b5b0-db336a33f67f"). InnerVolumeSpecName "kube-api-access-gqx6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:33:15.333749 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.333712    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c562aeb1-976f-4b17-b5b0-db336a33f67f-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:33:15.333749 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:15.333742    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqx6s\" (UniqueName: \"kubernetes.io/projected/c562aeb1-976f-4b17-b5b0-db336a33f67f-kube-api-access-gqx6s\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:33:16.067963 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:16.067933    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"
Oct 30 23:33:16.068406 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:16.067972    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9" event={"ID":"c562aeb1-976f-4b17-b5b0-db336a33f67f","Type":"ContainerDied","Data":"98c98c4104c31e856e9a184307858b02611256cefe0fd84415ace8bf1d4f0c44"}
Oct 30 23:33:16.068406 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:16.068038    5078 scope.go:122] "RemoveContainer" containerID="21cbe81217a410639e8d43b145ceb94887312bf8e34cd2d238bab3e9cfb8976f"
Oct 30 23:33:16.284591 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:16.284559    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"]
Oct 30 23:33:16.333910 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:16.333845    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-40c57b52-openshift-default-5ddcfb77cc-9zjw9"]
Oct 30 23:33:17.594483 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:33:17.594449    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:33:18.301273 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:18.301241    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c562aeb1-976f-4b17-b5b0-db336a33f67f" path="/var/lib/kubelet/pods/c562aeb1-976f-4b17-b5b0-db336a33f67f/volumes"
Oct 30 23:33:21.047342 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:21.047310    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96"]
Oct 30 23:33:21.096470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:21.096433    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07aadec7-7mm96"]
Oct 30 23:33:22.300103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.300068    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad2159d7-8592-4372-830f-e791ce996831" path="/var/lib/kubelet/pods/ad2159d7-8592-4372-830f-e791ce996831/volumes"
Oct 30 23:33:22.784225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.784151    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"]
Oct 30 23:33:22.784830 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.784815    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c562aeb1-976f-4b17-b5b0-db336a33f67f" containerName="istio-proxy"
Oct 30 23:33:22.784888 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.784831    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c562aeb1-976f-4b17-b5b0-db336a33f67f" containerName="istio-proxy"
Oct 30 23:33:22.784961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.784952    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c562aeb1-976f-4b17-b5b0-db336a33f67f" containerName="istio-proxy"
Oct 30 23:33:22.784997 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.784966    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c562aeb1-976f-4b17-b5b0-db336a33f67f" containerName="istio-proxy"
Oct 30 23:33:22.790186 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.790165    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:22.867580 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.867539    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:22.867580 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.867588    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:22.867796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.867635    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:22.867835 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.867816    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:22.867883 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.867859    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:22.868027 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.868004    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfqpv\" (UniqueName: \"kubernetes.io/projected/dc5a135a-f306-48a0-929e-3d9e44f97e1f-kube-api-access-lfqpv\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:22.868087 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.868071    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:22.868190 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.868170    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:22.868259 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.868231    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:22.891813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.891780    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"]
Oct 30 23:33:22.956674 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.936848    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-55666687-openshift-default-dockercfg-8kmlb\""
Oct 30 23:33:22.981056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.970147    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:23.055102 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:23.055072    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:23.055355 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:23.055336    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:23.055628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:23.055582    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:23.166619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:23.166532    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:23.166619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:23.111544    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:23.166878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:22.988081    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:23.273677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:23.176759    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfqpv\" (UniqueName: \"kubernetes.io/projected/dc5a135a-f306-48a0-929e-3d9e44f97e1f-kube-api-access-lfqpv\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:23.273677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:23.176843    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:23.273677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:23.176903    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:23.273677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:23.176966    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:23.273677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:23.178192    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:23.273677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:23.178817    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:23.273677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:23.179409    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:23.279617 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:23.274871    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfqpv\" (UniqueName: \"kubernetes.io/projected/dc5a135a-f306-48a0-929e-3d9e44f97e1f-kube-api-access-lfqpv\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:23.286655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:23.285625    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:23.325441 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:23.307963    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:23.325441 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:23.308570    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-77jfn\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:23.433623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:23.432037    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:24.008727 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:33:24.008697    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5a135a_f306_48a0_929e_3d9e44f97e1f.slice/crio-c73e96664c27b91a2590677cbe13fe98fd4e196eff6d1641fbbf609923bd7244 WatchSource:0}: Error finding container c73e96664c27b91a2590677cbe13fe98fd4e196eff6d1641fbbf609923bd7244: Status 404 returned error can't find the container with id c73e96664c27b91a2590677cbe13fe98fd4e196eff6d1641fbbf609923bd7244
Oct 30 23:33:24.008852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:24.008782    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"]
Oct 30 23:33:24.011135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:24.011104    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:33:24.011235 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:24.011165    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:33:24.011235 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:24.011192    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:33:24.242205 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:24.242166    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn" event={"ID":"dc5a135a-f306-48a0-929e-3d9e44f97e1f","Type":"ContainerStarted","Data":"a97256900f72473dae2680bdd03f16d45793d684590c300bfe40fe3927a38e46"}
Oct 30 23:33:24.242402 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:24.242228    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn" event={"ID":"dc5a135a-f306-48a0-929e-3d9e44f97e1f","Type":"ContainerStarted","Data":"c73e96664c27b91a2590677cbe13fe98fd4e196eff6d1641fbbf609923bd7244"}
Oct 30 23:33:24.362624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:24.361585    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn" podStartSLOduration=2.3615713879999998 podStartE2EDuration="2.361571388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:33:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:33:24.359047444 +0000 UTC m=+1590.246953822" watchObservedRunningTime="2026-10-30 23:33:24.361571388 +0000 UTC m=+1590.249477741"
Oct 30 23:33:24.432911 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:24.432867    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:25.056357 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:25.056323    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw"]
Oct 30 23:33:25.062447 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:25.062419    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-752f525b-8wllw"]
Oct 30 23:33:25.437911 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:25.437834    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:26.258722 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:26.258687    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:26.259610 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:26.259577    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:33:26.300773 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:26.300744    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11406c63-9c89-460f-bb55-1af3b29bacda" path="/var/lib/kubelet/pods/11406c63-9c89-460f-bb55-1af3b29bacda/volumes"
Oct 30 23:33:30.674332 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:30.674294    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp"]
Oct 30 23:33:30.674969 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:30.674955    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c562aeb1-976f-4b17-b5b0-db336a33f67f" containerName="istio-proxy"
Oct 30 23:33:30.675016 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:30.674970    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c562aeb1-976f-4b17-b5b0-db336a33f67f" containerName="istio-proxy"
Oct 30 23:33:30.678968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:30.678952    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp"
Oct 30 23:33:30.737644 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:30.737615    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp"]
Oct 30 23:33:30.781468 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:30.781433    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:33:30.781662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:30.781481    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:33:30.781662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:30.781624    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:33:30.781662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:30.781644    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:33:30.868272 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:30.868231    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxwc7\" (UniqueName: \"kubernetes.io/projected/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-kube-api-access-zxwc7\") pod \"maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp\" (UID: \"05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp"
Oct 30 23:33:30.868448 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:30.868422    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp\" (UID: \"05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp"
Oct 30 23:33:30.868783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:30.868721    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp\" (UID: \"05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp"
Oct 30 23:33:30.970059 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:30.969978    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp\" (UID: \"05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp"
Oct 30 23:33:30.970218 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:30.970089    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxwc7\" (UniqueName: \"kubernetes.io/projected/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-kube-api-access-zxwc7\") pod \"maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp\" (UID: \"05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp"
Oct 30 23:33:30.970218 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:30.970171    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp\" (UID: \"05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp"
Oct 30 23:33:30.970639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:30.970576    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp\" (UID: \"05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp"
Oct 30 23:33:30.972680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:30.972657    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp\" (UID: \"05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp"
Oct 30 23:33:31.049507 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:31.049436    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxwc7\" (UniqueName: \"kubernetes.io/projected/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-kube-api-access-zxwc7\") pod \"maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp\" (UID: \"05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp"
Oct 30 23:33:31.293990 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:31.293965    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp"
Oct 30 23:33:31.775462 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:31.775431    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp"]
Oct 30 23:33:31.780697 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:33:31.780649    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b1c0ab_aee3_4ebb_a9e8_96db5a6dabb4.slice/crio-21805cd573d24ebda84e589b9d3f7eaea5836d1f7575bbb2912874aecb847073 WatchSource:0}: Error finding container 21805cd573d24ebda84e589b9d3f7eaea5836d1f7575bbb2912874aecb847073: Status 404 returned error can't find the container with id 21805cd573d24ebda84e589b9d3f7eaea5836d1f7575bbb2912874aecb847073
Oct 30 23:33:31.913623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:31.896653    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v"]
Oct 30 23:33:31.930658 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:31.925421    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v"
Oct 30 23:33:32.015310 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:31.991151    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-55666687-dockercfg-95f85\""
Oct 30 23:33:32.015310 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:31.992306    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-55666687\""
Oct 30 23:33:32.248622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.247954    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj64g\" (UniqueName: \"kubernetes.io/projected/53db4c77-5707-4260-8d1b-83d85e0d9bc1-kube-api-access-kj64g\") pod \"payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v\" (UID: \"53db4c77-5707-4260-8d1b-83d85e0d9bc1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v"
Oct 30 23:33:32.248622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.248174    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53db4c77-5707-4260-8d1b-83d85e0d9bc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v\" (UID: \"53db4c77-5707-4260-8d1b-83d85e0d9bc1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v"
Oct 30 23:33:32.276624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.260008    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v"]
Oct 30 23:33:32.369617 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.359890    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj64g\" (UniqueName: \"kubernetes.io/projected/53db4c77-5707-4260-8d1b-83d85e0d9bc1-kube-api-access-kj64g\") pod \"payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v\" (UID: \"53db4c77-5707-4260-8d1b-83d85e0d9bc1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v"
Oct 30 23:33:32.370926 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.370897    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53db4c77-5707-4260-8d1b-83d85e0d9bc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v\" (UID: \"53db4c77-5707-4260-8d1b-83d85e0d9bc1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v"
Oct 30 23:33:32.377749 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.372201    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53db4c77-5707-4260-8d1b-83d85e0d9bc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v\" (UID: \"53db4c77-5707-4260-8d1b-83d85e0d9bc1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v"
Oct 30 23:33:32.410094 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.410070    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml"]
Oct 30 23:33:32.420039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.420015    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml"
Oct 30 23:33:32.421872 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.421847    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp" event={"ID":"05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4","Type":"ContainerStarted","Data":"21805cd573d24ebda84e589b9d3f7eaea5836d1f7575bbb2912874aecb847073"}
Oct 30 23:33:32.427062 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.427042    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml"]
Oct 30 23:33:32.428101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.428083    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj64g\" (UniqueName: \"kubernetes.io/projected/53db4c77-5707-4260-8d1b-83d85e0d9bc1-kube-api-access-kj64g\") pod \"payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v\" (UID: \"53db4c77-5707-4260-8d1b-83d85e0d9bc1\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v"
Oct 30 23:33:32.576332 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.576308    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjjkc\" (UniqueName: \"kubernetes.io/projected/92708dca-2c92-4ca1-b91d-57c81b176c79-kube-api-access-zjjkc\") pod \"payload-processing-e2e-mt-55666687-cdc987d69-2tdml\" (UID: \"92708dca-2c92-4ca1-b91d-57c81b176c79\") " pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml"
Oct 30 23:33:32.576443 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.576427    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92708dca-2c92-4ca1-b91d-57c81b176c79-plugins-config-volume\") pod \"payload-processing-e2e-mt-55666687-cdc987d69-2tdml\" (UID: \"92708dca-2c92-4ca1-b91d-57c81b176c79\") " pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml"
Oct 30 23:33:32.678275 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.678187    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjjkc\" (UniqueName: \"kubernetes.io/projected/92708dca-2c92-4ca1-b91d-57c81b176c79-kube-api-access-zjjkc\") pod \"payload-processing-e2e-mt-55666687-cdc987d69-2tdml\" (UID: \"92708dca-2c92-4ca1-b91d-57c81b176c79\") " pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml"
Oct 30 23:33:32.678449 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.678307    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92708dca-2c92-4ca1-b91d-57c81b176c79-plugins-config-volume\") pod \"payload-processing-e2e-mt-55666687-cdc987d69-2tdml\" (UID: \"92708dca-2c92-4ca1-b91d-57c81b176c79\") " pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml"
Oct 30 23:33:32.678992 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.678971    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92708dca-2c92-4ca1-b91d-57c81b176c79-plugins-config-volume\") pod \"payload-processing-e2e-mt-55666687-cdc987d69-2tdml\" (UID: \"92708dca-2c92-4ca1-b91d-57c81b176c79\") " pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml"
Oct 30 23:33:32.706348 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.706323    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v"
Oct 30 23:33:32.745444 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.745418    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjjkc\" (UniqueName: \"kubernetes.io/projected/92708dca-2c92-4ca1-b91d-57c81b176c79-kube-api-access-zjjkc\") pod \"payload-processing-e2e-mt-55666687-cdc987d69-2tdml\" (UID: \"92708dca-2c92-4ca1-b91d-57c81b176c79\") " pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml"
Oct 30 23:33:32.747354 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:32.747331    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml"
Oct 30 23:33:33.091644 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:33:33.091610    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53db4c77_5707_4260_8d1b_83d85e0d9bc1.slice/crio-75f65366d100f335e388044a0ec73efb7c491dcf58517538915cea32ba7ca169 WatchSource:0}: Error finding container 75f65366d100f335e388044a0ec73efb7c491dcf58517538915cea32ba7ca169: Status 404 returned error can't find the container with id 75f65366d100f335e388044a0ec73efb7c491dcf58517538915cea32ba7ca169
Oct 30 23:33:33.092284 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:33.092229    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v"]
Oct 30 23:33:33.147991 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:33.147967    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml"]
Oct 30 23:33:33.154444 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:33:33.154419    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92708dca_2c92_4ca1_b91d_57c81b176c79.slice/crio-d0cd04cb8e1a206ff53b2cc5266b8b274b80439a6e0aa62cb57bc71440af8f17 WatchSource:0}: Error finding container d0cd04cb8e1a206ff53b2cc5266b8b274b80439a6e0aa62cb57bc71440af8f17: Status 404 returned error can't find the container with id d0cd04cb8e1a206ff53b2cc5266b8b274b80439a6e0aa62cb57bc71440af8f17
Oct 30 23:33:33.431877 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:33.431801    5078 generic.go:356] "Generic (PLEG): container finished" podID="05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4" containerID="68e56b01fdece1a29508789274a81a82d6e5473ff570b7a30ff1714b2b43f6da" exitCode=0
Oct 30 23:33:33.432018 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:33.431882    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp" event={"ID":"05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4","Type":"ContainerDied","Data":"68e56b01fdece1a29508789274a81a82d6e5473ff570b7a30ff1714b2b43f6da"}
Oct 30 23:33:33.433690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:33.433665    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml" event={"ID":"92708dca-2c92-4ca1-b91d-57c81b176c79","Type":"ContainerStarted","Data":"57a138c7819944198154c009d090014e5a858cd6220c8ad63075e44405dae07f"}
Oct 30 23:33:33.433851 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:33.433713    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml" event={"ID":"92708dca-2c92-4ca1-b91d-57c81b176c79","Type":"ContainerStarted","Data":"d0cd04cb8e1a206ff53b2cc5266b8b274b80439a6e0aa62cb57bc71440af8f17"}
Oct 30 23:33:33.433851 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:33.433740    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml"
Oct 30 23:33:33.435382 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:33.435359    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v" event={"ID":"53db4c77-5707-4260-8d1b-83d85e0d9bc1","Type":"ContainerStarted","Data":"4846ea023d1a1046038706f2c241942ea0829375ee13b369db54be8704caf7c7"}
Oct 30 23:33:33.435556 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:33.435396    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v" event={"ID":"53db4c77-5707-4260-8d1b-83d85e0d9bc1","Type":"ContainerStarted","Data":"75f65366d100f335e388044a0ec73efb7c491dcf58517538915cea32ba7ca169"}
Oct 30 23:33:33.435556 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:33.435447    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v"
Oct 30 23:33:33.704391 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:33.704286    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml" podStartSLOduration=2.704272128 podStartE2EDuration="2.704272128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:33:33.701766067 +0000 UTC m=+1599.589672432" watchObservedRunningTime="2026-10-30 23:33:33.704272128 +0000 UTC m=+1599.592178480"
Oct 30 23:33:33.861625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:33.861430    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v" podStartSLOduration=2.861410849 podStartE2EDuration="2.861410849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:33:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:33:33.822366524 +0000 UTC m=+1599.710272877" watchObservedRunningTime="2026-10-30 23:33:33.861410849 +0000 UTC m=+1599.749317204"
Oct 30 23:33:34.603178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:34.603155    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp"
Oct 30 23:33:34.656045 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:34.656019    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-kube-api-access-zxwc7\" (UniqueName: \"kubernetes.io/projected/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-kube-api-access-zxwc7\") pod \"05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4\" (UID: \"05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4\") "
Oct 30 23:33:34.656186 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:34.656136    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-maas-api-xks-ca\") pod \"05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4\" (UID: \"05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4\") "
Oct 30 23:33:34.656269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:34.656214    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-maas-api-openshift-ca\") pod \"05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4\" (UID: \"05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4\") "
Oct 30 23:33:34.656585 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:34.656555    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-maas-api-openshift-ca" pod "05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4" (UID: "05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:33:34.656912 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:34.656857    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:33:34.658324 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:34.658300    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-maas-api-xks-ca" pod "05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4" (UID: "05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:33:34.658324 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:34.658311    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-kube-api-access-zxwc7" pod "05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4" (UID: "05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4"). InnerVolumeSpecName "kube-api-access-zxwc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:33:34.758043 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:34.757968    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:33:34.758043 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:34.758000    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxwc7\" (UniqueName: \"kubernetes.io/projected/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4-kube-api-access-zxwc7\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:33:35.452555 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:35.452525    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp"
Oct 30 23:33:35.452738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:35.452556    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp" event={"ID":"05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4","Type":"ContainerDied","Data":"21805cd573d24ebda84e589b9d3f7eaea5836d1f7575bbb2912874aecb847073"}
Oct 30 23:33:35.452738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:35.452636    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21805cd573d24ebda84e589b9d3f7eaea5836d1f7575bbb2912874aecb847073"
Oct 30 23:33:37.071947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:37.071912    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd"]
Oct 30 23:33:37.101077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:37.101016    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a89059e-sqbxd"]
Oct 30 23:33:38.049176 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:38.049142    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf"]
Oct 30 23:33:38.098427 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:38.098392    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-576b45a7-bjmvf"]
Oct 30 23:33:38.304306 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:38.304216    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b0a87d8-687d-4ebf-bd66-aa6cc59701b0" path="/var/lib/kubelet/pods/8b0a87d8-687d-4ebf-bd66-aa6cc59701b0/volumes"
Oct 30 23:33:38.305018 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:38.304991    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8f824de-6ac1-491a-abc3-d4a941873bdf" path="/var/lib/kubelet/pods/b8f824de-6ac1-491a-abc3-d4a941873bdf/volumes"
Oct 30 23:33:39.047977 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:39.047943    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs"]
Oct 30 23:33:39.097865 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:39.097832    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d88717da-tkgzs"]
Oct 30 23:33:40.301157 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:40.301121    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="496633ba-dc94-4a3c-a6c4-079a2d0234b9" path="/var/lib/kubelet/pods/496633ba-dc94-4a3c-a6c4-079a2d0234b9/volumes"
Oct 30 23:33:42.193453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:42.193419    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn"]
Oct 30 23:33:42.302477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:42.302446    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n"]
Oct 30 23:33:44.445948 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:44.445921    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v"
Oct 30 23:33:44.446356 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:44.445996    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml"
Oct 30 23:33:47.194262 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.194184    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn" podUID="4ea6ae1c-d046-499e-89c6-22a626aa98c4" containerName="payload-processing" containerID="cri-o://de28fe341a7e895b3f183626864c534aa6887603bb6bd9e537f80a3d40b72ff5" gracePeriod=25
Oct 30 23:33:47.303315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.303277    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n" podUID="0471065d-e2f3-4d32-83df-90c783556e1d" containerName="payload-pre-processing" containerID="cri-o://33c69b6b32a704f9fbd7798a33f0a4c80740ea80e90aa238648bf7e461e7102b" gracePeriod=25
Oct 30 23:33:47.480052 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.480022    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn"
Oct 30 23:33:47.589178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.589146    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea6ae1c-d046-499e-89c6-22a626aa98c4-kube-api-access-xsm5g\" (UniqueName: \"kubernetes.io/projected/4ea6ae1c-d046-499e-89c6-22a626aa98c4-kube-api-access-xsm5g\") pod \"4ea6ae1c-d046-499e-89c6-22a626aa98c4\" (UID: \"4ea6ae1c-d046-499e-89c6-22a626aa98c4\") "
Oct 30 23:33:47.589338 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.589234    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ea6ae1c-d046-499e-89c6-22a626aa98c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea6ae1c-d046-499e-89c6-22a626aa98c4-plugins-config-volume\") pod \"4ea6ae1c-d046-499e-89c6-22a626aa98c4\" (UID: \"4ea6ae1c-d046-499e-89c6-22a626aa98c4\") "
Oct 30 23:33:47.589907 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.589878    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea6ae1c-d046-499e-89c6-22a626aa98c4-plugins-config-volume" pod "4ea6ae1c-d046-499e-89c6-22a626aa98c4" (UID: "4ea6ae1c-d046-499e-89c6-22a626aa98c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:33:47.591871 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.591842    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea6ae1c-d046-499e-89c6-22a626aa98c4-kube-api-access-xsm5g" pod "4ea6ae1c-d046-499e-89c6-22a626aa98c4" (UID: "4ea6ae1c-d046-499e-89c6-22a626aa98c4"). InnerVolumeSpecName "kube-api-access-xsm5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:33:47.608463 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:33:47.608436    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:33:47.622055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.622030    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n"
Oct 30 23:33:47.634082 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.634051    5078 generic.go:356] "Generic (PLEG): container finished" podID="0471065d-e2f3-4d32-83df-90c783556e1d" containerID="33c69b6b32a704f9fbd7798a33f0a4c80740ea80e90aa238648bf7e461e7102b" exitCode=0
Oct 30 23:33:47.634219 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.634106    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n"
Oct 30 23:33:47.634219 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.634159    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n" event={"ID":"0471065d-e2f3-4d32-83df-90c783556e1d","Type":"ContainerDied","Data":"33c69b6b32a704f9fbd7798a33f0a4c80740ea80e90aa238648bf7e461e7102b"}
Oct 30 23:33:47.634335 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.634276    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n" event={"ID":"0471065d-e2f3-4d32-83df-90c783556e1d","Type":"ContainerDied","Data":"64799e019bf5d63d52a51efc3d93325273caaac5b784221920af5d74458ce531"}
Oct 30 23:33:47.634335 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.634312    5078 scope.go:122] "RemoveContainer" containerID="33c69b6b32a704f9fbd7798a33f0a4c80740ea80e90aa238648bf7e461e7102b"
Oct 30 23:33:47.635749 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.635728    5078 generic.go:356] "Generic (PLEG): container finished" podID="4ea6ae1c-d046-499e-89c6-22a626aa98c4" containerID="de28fe341a7e895b3f183626864c534aa6887603bb6bd9e537f80a3d40b72ff5" exitCode=0
Oct 30 23:33:47.635854 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.635757    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn" event={"ID":"4ea6ae1c-d046-499e-89c6-22a626aa98c4","Type":"ContainerDied","Data":"de28fe341a7e895b3f183626864c534aa6887603bb6bd9e537f80a3d40b72ff5"}
Oct 30 23:33:47.635854 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.635828    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn" event={"ID":"4ea6ae1c-d046-499e-89c6-22a626aa98c4","Type":"ContainerDied","Data":"eb81afea1249465e094e3409be6e73e7153220c1bdd9d8e6732b794d29641ba9"}
Oct 30 23:33:47.635854 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.635776    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn"
Oct 30 23:33:47.645259 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.645174    5078 scope.go:122] "RemoveContainer" containerID="33c69b6b32a704f9fbd7798a33f0a4c80740ea80e90aa238648bf7e461e7102b"
Oct 30 23:33:47.645507 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:33:47.645483    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c69b6b32a704f9fbd7798a33f0a4c80740ea80e90aa238648bf7e461e7102b\": container with ID starting with 33c69b6b32a704f9fbd7798a33f0a4c80740ea80e90aa238648bf7e461e7102b not found: ID does not exist" containerID="33c69b6b32a704f9fbd7798a33f0a4c80740ea80e90aa238648bf7e461e7102b"
Oct 30 23:33:47.645568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.645522    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c69b6b32a704f9fbd7798a33f0a4c80740ea80e90aa238648bf7e461e7102b"} err="failed to get container status \"33c69b6b32a704f9fbd7798a33f0a4c80740ea80e90aa238648bf7e461e7102b\": rpc error: code = NotFound desc = could not find container \"33c69b6b32a704f9fbd7798a33f0a4c80740ea80e90aa238648bf7e461e7102b\": container with ID starting with 33c69b6b32a704f9fbd7798a33f0a4c80740ea80e90aa238648bf7e461e7102b not found: ID does not exist"
Oct 30 23:33:47.645568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.645547    5078 scope.go:122] "RemoveContainer" containerID="de28fe341a7e895b3f183626864c534aa6887603bb6bd9e537f80a3d40b72ff5"
Oct 30 23:33:47.657302 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.657282    5078 scope.go:122] "RemoveContainer" containerID="de28fe341a7e895b3f183626864c534aa6887603bb6bd9e537f80a3d40b72ff5"
Oct 30 23:33:47.657614 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:33:47.657577    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de28fe341a7e895b3f183626864c534aa6887603bb6bd9e537f80a3d40b72ff5\": container with ID starting with de28fe341a7e895b3f183626864c534aa6887603bb6bd9e537f80a3d40b72ff5 not found: ID does not exist" containerID="de28fe341a7e895b3f183626864c534aa6887603bb6bd9e537f80a3d40b72ff5"
Oct 30 23:33:47.657720 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.657669    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de28fe341a7e895b3f183626864c534aa6887603bb6bd9e537f80a3d40b72ff5"} err="failed to get container status \"de28fe341a7e895b3f183626864c534aa6887603bb6bd9e537f80a3d40b72ff5\": rpc error: code = NotFound desc = could not find container \"de28fe341a7e895b3f183626864c534aa6887603bb6bd9e537f80a3d40b72ff5\": container with ID starting with de28fe341a7e895b3f183626864c534aa6887603bb6bd9e537f80a3d40b72ff5 not found: ID does not exist"
Oct 30 23:33:47.691144 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.691123    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0471065d-e2f3-4d32-83df-90c783556e1d-kube-api-access-8q279\" (UniqueName: \"kubernetes.io/projected/0471065d-e2f3-4d32-83df-90c783556e1d-kube-api-access-8q279\") pod \"0471065d-e2f3-4d32-83df-90c783556e1d\" (UID: \"0471065d-e2f3-4d32-83df-90c783556e1d\") "
Oct 30 23:33:47.691305 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.691293    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0471065d-e2f3-4d32-83df-90c783556e1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0471065d-e2f3-4d32-83df-90c783556e1d-plugins-config-volume\") pod \"0471065d-e2f3-4d32-83df-90c783556e1d\" (UID: \"0471065d-e2f3-4d32-83df-90c783556e1d\") "
Oct 30 23:33:47.691675 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.691646    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0471065d-e2f3-4d32-83df-90c783556e1d-plugins-config-volume" pod "0471065d-e2f3-4d32-83df-90c783556e1d" (UID: "0471065d-e2f3-4d32-83df-90c783556e1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:33:47.691886 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.691870    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0471065d-e2f3-4d32-83df-90c783556e1d-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:33:47.691958 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.691888    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsm5g\" (UniqueName: \"kubernetes.io/projected/4ea6ae1c-d046-499e-89c6-22a626aa98c4-kube-api-access-xsm5g\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:33:47.691958 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.691898    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ea6ae1c-d046-499e-89c6-22a626aa98c4-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:33:47.693216 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.693197    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0471065d-e2f3-4d32-83df-90c783556e1d-kube-api-access-8q279" pod "0471065d-e2f3-4d32-83df-90c783556e1d" (UID: "0471065d-e2f3-4d32-83df-90c783556e1d"). InnerVolumeSpecName "kube-api-access-8q279". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:33:47.793028 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.793000    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q279\" (UniqueName: \"kubernetes.io/projected/0471065d-e2f3-4d32-83df-90c783556e1d-kube-api-access-8q279\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:33:47.903149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.903117    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn"]
Oct 30 23:33:47.907355 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:47.907334    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-5f9ec6be-657f587f4f-szpdn"]
Oct 30 23:33:48.050695 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:48.050590    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px"]
Oct 30 23:33:48.056322 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:48.056293    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc"]
Oct 30 23:33:48.062576 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:48.062555    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-538ff61a-8h6px"]
Oct 30 23:33:48.068650 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:48.068623    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n"]
Oct 30 23:33:48.072221 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:48.072199    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e1c4370-8z7kc"]
Oct 30 23:33:48.099481 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:48.099451    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-5f9ec6be-cd9f5cb47-dpj8n"]
Oct 30 23:33:48.300755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:48.300684    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0471065d-e2f3-4d32-83df-90c783556e1d" path="/var/lib/kubelet/pods/0471065d-e2f3-4d32-83df-90c783556e1d/volumes"
Oct 30 23:33:48.301118 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:48.301068    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea6ae1c-d046-499e-89c6-22a626aa98c4" path="/var/lib/kubelet/pods/4ea6ae1c-d046-499e-89c6-22a626aa98c4/volumes"
Oct 30 23:33:48.301470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:48.301457    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6094da88-cad0-4909-81be-8bb07fb4154e" path="/var/lib/kubelet/pods/6094da88-cad0-4909-81be-8bb07fb4154e/volumes"
Oct 30 23:33:48.302172 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:48.302158    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb93fbfc-171e-44bc-867c-25e3cfdd883a" path="/var/lib/kubelet/pods/fb93fbfc-171e-44bc-867c-25e3cfdd883a/volumes"
Oct 30 23:33:49.054422 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:49.054382    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj"]
Oct 30 23:33:49.104037 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:49.104007    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8785bf53-2r7mj"]
Oct 30 23:33:49.616397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:49.616364    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp"]
Oct 30 23:33:49.773626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:49.773428    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5f9ec6be-r7nlp"]
Oct 30 23:33:50.300491 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:50.300457    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4" path="/var/lib/kubelet/pods/05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4/volumes"
Oct 30 23:33:50.300982 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:50.300965    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1deebc5-fa99-4e05-9ca2-6122a1fa20bc" path="/var/lib/kubelet/pods/d1deebc5-fa99-4e05-9ca2-6122a1fa20bc/volumes"
Oct 30 23:33:54.048819 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:54.048785    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5"]
Oct 30 23:33:54.098516 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:54.098487    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2371acca-r5mb5"]
Oct 30 23:33:54.300777 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:54.300706    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2a09de6-56f1-4729-899d-3a0929e954f4" path="/var/lib/kubelet/pods/b2a09de6-56f1-4729-899d-3a0929e954f4/volumes"
Oct 30 23:33:56.238762 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:56.238717    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"]
Oct 30 23:33:56.239248 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:56.238970    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7" podUID="d511dc2f-93c4-41f7-a6a2-3def48b402b4" containerName="istio-proxy" containerID="cri-o://b088dcff896b0b2048897f9d0ec35a55a9669c71507e4efc5a95061a756094c4" gracePeriod=30
Oct 30 23:33:58.047826 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:58.047795    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2"]
Oct 30 23:33:58.097430 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:58.097401    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-194230bb-5ndw2"]
Oct 30 23:33:58.299502 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:33:58.299427    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9e5057f-96ec-4a37-a54c-6c15c320283b" path="/var/lib/kubelet/pods/c9e5057f-96ec-4a37-a54c-6c15c320283b/volumes"
Oct 30 23:34:00.702449 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:00.702413    5078 scope.go:122] "RemoveContainer" containerID="e628021035c74b5ab1dc6f0c8a0fc0d49450786f94ba144defb1c4ef1edcddbb"
Oct 30 23:34:00.713116 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:00.713096    5078 scope.go:122] "RemoveContainer" containerID="2dec40a2a0b46f93bd3f217b176f452572378844593884fda70b9e9fb523f1af"
Oct 30 23:34:00.723904 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:00.723885    5078 scope.go:122] "RemoveContainer" containerID="07df542f8d6055b8be34fe9e8ef102a552fd537ad5207c31dce8ef73757b2032"
Oct 30 23:34:00.733244 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:00.733224    5078 scope.go:122] "RemoveContainer" containerID="9ee54cedf4c019a27949e275c84c25512171fb3aee8b1a4a662c5f4bfe894167"
Oct 30 23:34:01.544004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.543977    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:34:01.549327 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.549300    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d"]
Oct 30 23:34:01.550011 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.549996    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0471065d-e2f3-4d32-83df-90c783556e1d" containerName="payload-pre-processing"
Oct 30 23:34:01.550056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.550013    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0471065d-e2f3-4d32-83df-90c783556e1d" containerName="payload-pre-processing"
Oct 30 23:34:01.550056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.550023    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d511dc2f-93c4-41f7-a6a2-3def48b402b4" containerName="istio-proxy"
Oct 30 23:34:01.550056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.550028    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d511dc2f-93c4-41f7-a6a2-3def48b402b4" containerName="istio-proxy"
Oct 30 23:34:01.550056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.550036    5078 cpu_manager.go:414] "containerMap: removing container" podUID="05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4" containerName="revoke-keys"
Oct 30 23:34:01.550056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.550041    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4" containerName="revoke-keys"
Oct 30 23:34:01.550056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.550048    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d511dc2f-93c4-41f7-a6a2-3def48b402b4" containerName="istio-proxy"
Oct 30 23:34:01.550056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.550054    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d511dc2f-93c4-41f7-a6a2-3def48b402b4" containerName="istio-proxy"
Oct 30 23:34:01.550279 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.550062    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4ea6ae1c-d046-499e-89c6-22a626aa98c4" containerName="payload-processing"
Oct 30 23:34:01.550279 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.550067    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea6ae1c-d046-499e-89c6-22a626aa98c4" containerName="payload-processing"
Oct 30 23:34:01.550279 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.550164    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0471065d-e2f3-4d32-83df-90c783556e1d" containerName="payload-pre-processing"
Oct 30 23:34:01.550279 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.550170    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d511dc2f-93c4-41f7-a6a2-3def48b402b4" containerName="istio-proxy"
Oct 30 23:34:01.550279 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.550175    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="05b1c0ab-aee3-4ebb-a9e8-96db5a6dabb4" containerName="revoke-keys"
Oct 30 23:34:01.550279 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.550180    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d511dc2f-93c4-41f7-a6a2-3def48b402b4" containerName="istio-proxy"
Oct 30 23:34:01.550279 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.550186    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea6ae1c-d046-499e-89c6-22a626aa98c4" containerName="payload-processing"
Oct 30 23:34:01.553632 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.553612    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d"
Oct 30 23:34:01.629998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.629959    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjwmg\" (UniqueName: \"kubernetes.io/projected/3afe0e20-7669-47b0-9ad4-6033155664a9-kube-api-access-wjwmg\") pod \"maas-api-revoke-sub-a283f3ba-nfm8d\" (UID: \"3afe0e20-7669-47b0-9ad4-6033155664a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d"
Oct 30 23:34:01.629998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.629997    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afe0e20-7669-47b0-9ad4-6033155664a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a283f3ba-nfm8d\" (UID: \"3afe0e20-7669-47b0-9ad4-6033155664a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d"
Oct 30 23:34:01.630411 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.630383    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afe0e20-7669-47b0-9ad4-6033155664a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a283f3ba-nfm8d\" (UID: \"3afe0e20-7669-47b0-9ad4-6033155664a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d"
Oct 30 23:34:01.646874 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.645814    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:34:01.646874 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.645814    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:34:01.646874 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.645817    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:34:01.646874 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.645854    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:34:01.650587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.650562    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d"]
Oct 30 23:34:01.731462 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.731430    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d511dc2f-93c4-41f7-a6a2-3def48b402b4-kube-api-access-hncpl\" (UniqueName: \"kubernetes.io/projected/d511dc2f-93c4-41f7-a6a2-3def48b402b4-kube-api-access-hncpl\") pod \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") "
Oct 30 23:34:01.731941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.731490    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istiod-ca-cert\") pod \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") "
Oct 30 23:34:01.731941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.731639    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-token\" (UniqueName: \"kubernetes.io/projected/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-token\") pod \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") "
Oct 30 23:34:01.731941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.731694    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-workload-certs\") pod \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") "
Oct 30 23:34:01.731941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.731724    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-workload-socket\") pod \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") "
Oct 30 23:34:01.731941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.731783    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-credential-socket\") pod \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") "
Oct 30 23:34:01.731941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.731817    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-podinfo\") pod \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") "
Oct 30 23:34:01.731941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.731863    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-envoy\") pod \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") "
Oct 30 23:34:01.731941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.731923    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-data\") pod \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\" (UID: \"d511dc2f-93c4-41f7-a6a2-3def48b402b4\") "
Oct 30 23:34:01.732410 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.731993    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-workload-certs" pod "d511dc2f-93c4-41f7-a6a2-3def48b402b4" (UID: "d511dc2f-93c4-41f7-a6a2-3def48b402b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:01.732410 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.732006    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-workload-socket" pod "d511dc2f-93c4-41f7-a6a2-3def48b402b4" (UID: "d511dc2f-93c4-41f7-a6a2-3def48b402b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:01.732410 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.732015    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istiod-ca-cert" pod "d511dc2f-93c4-41f7-a6a2-3def48b402b4" (UID: "d511dc2f-93c4-41f7-a6a2-3def48b402b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:34:01.732410 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.732088    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-credential-socket" pod "d511dc2f-93c4-41f7-a6a2-3def48b402b4" (UID: "d511dc2f-93c4-41f7-a6a2-3def48b402b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:01.732410 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.732316    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afe0e20-7669-47b0-9ad4-6033155664a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a283f3ba-nfm8d\" (UID: \"3afe0e20-7669-47b0-9ad4-6033155664a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d"
Oct 30 23:34:01.732410 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.732318    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-data" pod "d511dc2f-93c4-41f7-a6a2-3def48b402b4" (UID: "d511dc2f-93c4-41f7-a6a2-3def48b402b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:01.732726 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.732528    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjwmg\" (UniqueName: \"kubernetes.io/projected/3afe0e20-7669-47b0-9ad4-6033155664a9-kube-api-access-wjwmg\") pod \"maas-api-revoke-sub-a283f3ba-nfm8d\" (UID: \"3afe0e20-7669-47b0-9ad4-6033155664a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d"
Oct 30 23:34:01.732726 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.732572    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afe0e20-7669-47b0-9ad4-6033155664a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a283f3ba-nfm8d\" (UID: \"3afe0e20-7669-47b0-9ad4-6033155664a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d"
Oct 30 23:34:01.732852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.732823    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:01.733445 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.732847    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:01.733445 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.732959    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:01.733445 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.732978    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:01.733445 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.732994    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:01.733445 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.733106    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afe0e20-7669-47b0-9ad4-6033155664a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a283f3ba-nfm8d\" (UID: \"3afe0e20-7669-47b0-9ad4-6033155664a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d"
Oct 30 23:34:01.735042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.735009    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d511dc2f-93c4-41f7-a6a2-3def48b402b4-kube-api-access-hncpl" pod "d511dc2f-93c4-41f7-a6a2-3def48b402b4" (UID: "d511dc2f-93c4-41f7-a6a2-3def48b402b4"). InnerVolumeSpecName "kube-api-access-hncpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:34:01.735042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.735020    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-envoy" pod "d511dc2f-93c4-41f7-a6a2-3def48b402b4" (UID: "d511dc2f-93c4-41f7-a6a2-3def48b402b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:34:01.735255 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.735235    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-podinfo" pod "d511dc2f-93c4-41f7-a6a2-3def48b402b4" (UID: "d511dc2f-93c4-41f7-a6a2-3def48b402b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:34:01.735628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.735589    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-token" pod "d511dc2f-93c4-41f7-a6a2-3def48b402b4" (UID: "d511dc2f-93c4-41f7-a6a2-3def48b402b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:34:01.736037 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.736020    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afe0e20-7669-47b0-9ad4-6033155664a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a283f3ba-nfm8d\" (UID: \"3afe0e20-7669-47b0-9ad4-6033155664a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d"
Oct 30 23:34:01.790333 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.790293    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjwmg\" (UniqueName: \"kubernetes.io/projected/3afe0e20-7669-47b0-9ad4-6033155664a9-kube-api-access-wjwmg\") pod \"maas-api-revoke-sub-a283f3ba-nfm8d\" (UID: \"3afe0e20-7669-47b0-9ad4-6033155664a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d"
Oct 30 23:34:01.834774 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.834741    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:01.834774 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.834767    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:01.834774 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.834777    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d511dc2f-93c4-41f7-a6a2-3def48b402b4-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:01.835009 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.834787    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hncpl\" (UniqueName: \"kubernetes.io/projected/d511dc2f-93c4-41f7-a6a2-3def48b402b4-kube-api-access-hncpl\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:01.867175 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.867148    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d"
Oct 30 23:34:01.887147 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.887121    5078 generic.go:356] "Generic (PLEG): container finished" podID="d511dc2f-93c4-41f7-a6a2-3def48b402b4" containerID="b088dcff896b0b2048897f9d0ec35a55a9669c71507e4efc5a95061a756094c4" exitCode=0
Oct 30 23:34:01.887271 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.887177    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"
Oct 30 23:34:01.887271 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.887206    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7" event={"ID":"d511dc2f-93c4-41f7-a6a2-3def48b402b4","Type":"ContainerDied","Data":"b088dcff896b0b2048897f9d0ec35a55a9669c71507e4efc5a95061a756094c4"}
Oct 30 23:34:01.887392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.887273    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7" event={"ID":"d511dc2f-93c4-41f7-a6a2-3def48b402b4","Type":"ContainerDied","Data":"fa98391d9763d3b9eda364ec3cbad047e00e049730a8a7d4f7a30cc7edcfe732"}
Oct 30 23:34:01.887392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.887304    5078 scope.go:122] "RemoveContainer" containerID="b088dcff896b0b2048897f9d0ec35a55a9669c71507e4efc5a95061a756094c4"
Oct 30 23:34:01.899298 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.899278    5078 scope.go:122] "RemoveContainer" containerID="2235418a3fbb2c50be88a3744a178e7edca02abd8dcb0facab138a7b08c3d9f8"
Oct 30 23:34:01.913075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.913056    5078 scope.go:122] "RemoveContainer" containerID="b088dcff896b0b2048897f9d0ec35a55a9669c71507e4efc5a95061a756094c4"
Oct 30 23:34:01.913373 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:34:01.913351    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b088dcff896b0b2048897f9d0ec35a55a9669c71507e4efc5a95061a756094c4\": container with ID starting with b088dcff896b0b2048897f9d0ec35a55a9669c71507e4efc5a95061a756094c4 not found: ID does not exist" containerID="b088dcff896b0b2048897f9d0ec35a55a9669c71507e4efc5a95061a756094c4"
Oct 30 23:34:01.913429 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.913383    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b088dcff896b0b2048897f9d0ec35a55a9669c71507e4efc5a95061a756094c4"} err="failed to get container status \"b088dcff896b0b2048897f9d0ec35a55a9669c71507e4efc5a95061a756094c4\": rpc error: code = NotFound desc = could not find container \"b088dcff896b0b2048897f9d0ec35a55a9669c71507e4efc5a95061a756094c4\": container with ID starting with b088dcff896b0b2048897f9d0ec35a55a9669c71507e4efc5a95061a756094c4 not found: ID does not exist"
Oct 30 23:34:01.913429 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.913403    5078 scope.go:122] "RemoveContainer" containerID="2235418a3fbb2c50be88a3744a178e7edca02abd8dcb0facab138a7b08c3d9f8"
Oct 30 23:34:01.913666 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:34:01.913647    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2235418a3fbb2c50be88a3744a178e7edca02abd8dcb0facab138a7b08c3d9f8\": container with ID starting with 2235418a3fbb2c50be88a3744a178e7edca02abd8dcb0facab138a7b08c3d9f8 not found: ID does not exist" containerID="2235418a3fbb2c50be88a3744a178e7edca02abd8dcb0facab138a7b08c3d9f8"
Oct 30 23:34:01.913720 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:01.913670    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2235418a3fbb2c50be88a3744a178e7edca02abd8dcb0facab138a7b08c3d9f8"} err="failed to get container status \"2235418a3fbb2c50be88a3744a178e7edca02abd8dcb0facab138a7b08c3d9f8\": rpc error: code = NotFound desc = could not find container \"2235418a3fbb2c50be88a3744a178e7edca02abd8dcb0facab138a7b08c3d9f8\": container with ID starting with 2235418a3fbb2c50be88a3744a178e7edca02abd8dcb0facab138a7b08c3d9f8 not found: ID does not exist"
Oct 30 23:34:02.056704 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:02.056614    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"]
Oct 30 23:34:02.064438 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:02.064412    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-5f9ec6be-openshift-default-79bb8cd685-gnpt7"]
Oct 30 23:34:02.253072 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:02.253044    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d"]
Oct 30 23:34:02.253217 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:34:02.253109    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afe0e20_7669_47b0_9ad4_6033155664a9.slice/crio-95f3356bf89547e65c7cfd4efcd1f82fafb42758d570c1ebd9152b68a41fb22e WatchSource:0}: Error finding container 95f3356bf89547e65c7cfd4efcd1f82fafb42758d570c1ebd9152b68a41fb22e: Status 404 returned error can't find the container with id 95f3356bf89547e65c7cfd4efcd1f82fafb42758d570c1ebd9152b68a41fb22e
Oct 30 23:34:02.302620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:02.302569    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d511dc2f-93c4-41f7-a6a2-3def48b402b4" path="/var/lib/kubelet/pods/d511dc2f-93c4-41f7-a6a2-3def48b402b4/volumes"
Oct 30 23:34:02.896980 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:02.896887    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d" event={"ID":"3afe0e20-7669-47b0-9ad4-6033155664a9","Type":"ContainerStarted","Data":"99319e0f2c409812cda9889f6b3119dace4e6fe523d50f5d895944fb72a93334"}
Oct 30 23:34:02.897425 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:02.897042    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d" event={"ID":"3afe0e20-7669-47b0-9ad4-6033155664a9","Type":"ContainerStarted","Data":"95f3356bf89547e65c7cfd4efcd1f82fafb42758d570c1ebd9152b68a41fb22e"}
Oct 30 23:34:03.011830 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:03.011781    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d" podStartSLOduration=2.011767623 podStartE2EDuration="2.011767623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:34:03.01077415 +0000 UTC m=+1628.898680507" watchObservedRunningTime="2026-10-30 23:34:03.011767623 +0000 UTC m=+1628.899673976"
Oct 30 23:34:03.908848 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:03.908813    5078 generic.go:356] "Generic (PLEG): container finished" podID="3afe0e20-7669-47b0-9ad4-6033155664a9" containerID="99319e0f2c409812cda9889f6b3119dace4e6fe523d50f5d895944fb72a93334" exitCode=0
Oct 30 23:34:03.909287 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:03.908899    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d" event={"ID":"3afe0e20-7669-47b0-9ad4-6033155664a9","Type":"ContainerDied","Data":"99319e0f2c409812cda9889f6b3119dace4e6fe523d50f5d895944fb72a93334"}
Oct 30 23:34:05.134252 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:05.134227    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d"
Oct 30 23:34:05.197830 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:05.197803    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3afe0e20-7669-47b0-9ad4-6033155664a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afe0e20-7669-47b0-9ad4-6033155664a9-maas-api-openshift-ca\") pod \"3afe0e20-7669-47b0-9ad4-6033155664a9\" (UID: \"3afe0e20-7669-47b0-9ad4-6033155664a9\") "
Oct 30 23:34:05.198024 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:05.197841    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3afe0e20-7669-47b0-9ad4-6033155664a9-kube-api-access-wjwmg\" (UniqueName: \"kubernetes.io/projected/3afe0e20-7669-47b0-9ad4-6033155664a9-kube-api-access-wjwmg\") pod \"3afe0e20-7669-47b0-9ad4-6033155664a9\" (UID: \"3afe0e20-7669-47b0-9ad4-6033155664a9\") "
Oct 30 23:34:05.198024 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:05.197943    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3afe0e20-7669-47b0-9ad4-6033155664a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afe0e20-7669-47b0-9ad4-6033155664a9-maas-api-xks-ca\") pod \"3afe0e20-7669-47b0-9ad4-6033155664a9\" (UID: \"3afe0e20-7669-47b0-9ad4-6033155664a9\") "
Oct 30 23:34:05.198252 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:05.198222    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3afe0e20-7669-47b0-9ad4-6033155664a9-maas-api-openshift-ca" pod "3afe0e20-7669-47b0-9ad4-6033155664a9" (UID: "3afe0e20-7669-47b0-9ad4-6033155664a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:34:05.198795 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:05.198769    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3afe0e20-7669-47b0-9ad4-6033155664a9-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:05.200265 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:05.200231    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3afe0e20-7669-47b0-9ad4-6033155664a9-maas-api-xks-ca" pod "3afe0e20-7669-47b0-9ad4-6033155664a9" (UID: "3afe0e20-7669-47b0-9ad4-6033155664a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:34:05.200406 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:05.200387    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3afe0e20-7669-47b0-9ad4-6033155664a9-kube-api-access-wjwmg" pod "3afe0e20-7669-47b0-9ad4-6033155664a9" (UID: "3afe0e20-7669-47b0-9ad4-6033155664a9"). InnerVolumeSpecName "kube-api-access-wjwmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:34:05.300726 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:05.300693    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjwmg\" (UniqueName: \"kubernetes.io/projected/3afe0e20-7669-47b0-9ad4-6033155664a9-kube-api-access-wjwmg\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:05.300726 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:05.300729    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3afe0e20-7669-47b0-9ad4-6033155664a9-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:06.009621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:06.008985    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d" event={"ID":"3afe0e20-7669-47b0-9ad4-6033155664a9","Type":"ContainerDied","Data":"95f3356bf89547e65c7cfd4efcd1f82fafb42758d570c1ebd9152b68a41fb22e"}
Oct 30 23:34:06.009621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:06.009055    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95f3356bf89547e65c7cfd4efcd1f82fafb42758d570c1ebd9152b68a41fb22e"
Oct 30 23:34:06.009621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:06.009164    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d"
Oct 30 23:34:08.049808 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:08.049775    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm"]
Oct 30 23:34:08.099013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:08.098982    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f772e68d-c2wbm"]
Oct 30 23:34:08.299954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:08.299879    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32f88243-cb68-41f2-b187-79830eb3d662" path="/var/lib/kubelet/pods/32f88243-cb68-41f2-b187-79830eb3d662/volumes"
Oct 30 23:34:09.048299 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:09.048267    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h"]
Oct 30 23:34:09.097768 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:09.097737    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61525f73-jlx5h"]
Oct 30 23:34:10.299343 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:10.299314    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76ec7768-8590-4a51-8c3a-e47d659b0d56" path="/var/lib/kubelet/pods/76ec7768-8590-4a51-8c3a-e47d659b0d56/volumes"
Oct 30 23:34:11.046672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:11.046636    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs"]
Oct 30 23:34:11.095779 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:11.095753    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-564d5020-lswjs"]
Oct 30 23:34:12.047890 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.047861    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9"]
Oct 30 23:34:12.097541 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.097511    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-525485d7-p6rv9"]
Oct 30 23:34:12.301070 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.300986    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d12e01-68f8-431f-8ad7-69d6eb74bbbc" path="/var/lib/kubelet/pods/64d12e01-68f8-431f-8ad7-69d6eb74bbbc/volumes"
Oct 30 23:34:12.301864 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.301842    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8e48362-06af-40ad-8e4d-089ebcb9bb78" path="/var/lib/kubelet/pods/e8e48362-06af-40ad-8e4d-089ebcb9bb78/volumes"
Oct 30 23:34:12.458801 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.458770    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2"]
Oct 30 23:34:12.459446 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.459424    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3afe0e20-7669-47b0-9ad4-6033155664a9" containerName="revoke-keys"
Oct 30 23:34:12.459446 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.459443    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3afe0e20-7669-47b0-9ad4-6033155664a9" containerName="revoke-keys"
Oct 30 23:34:12.459725 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.459609    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3afe0e20-7669-47b0-9ad4-6033155664a9" containerName="revoke-keys"
Oct 30 23:34:12.465105 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.465087    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2"
Oct 30 23:34:12.522157 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.522131    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2"]
Oct 30 23:34:12.530159 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.530134    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33d1f453-d599-4b5a-8701-76670832838e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37084415-f9px2\" (UID: \"33d1f453-d599-4b5a-8701-76670832838e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2"
Oct 30 23:34:12.530304 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.530223    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33d1f453-d599-4b5a-8701-76670832838e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37084415-f9px2\" (UID: \"33d1f453-d599-4b5a-8701-76670832838e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2"
Oct 30 23:34:12.530304 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.530279    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5d4l\" (UniqueName: \"kubernetes.io/projected/33d1f453-d599-4b5a-8701-76670832838e-kube-api-access-f5d4l\") pod \"maas-api-revoke-sub-37084415-f9px2\" (UID: \"33d1f453-d599-4b5a-8701-76670832838e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2"
Oct 30 23:34:12.565405 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.565335    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:34:12.565577 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.565548    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:34:12.566724 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.566700    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:34:12.566845 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.566744    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:34:12.632099 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.632071    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33d1f453-d599-4b5a-8701-76670832838e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37084415-f9px2\" (UID: \"33d1f453-d599-4b5a-8701-76670832838e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2"
Oct 30 23:34:12.632255 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.632137    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33d1f453-d599-4b5a-8701-76670832838e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37084415-f9px2\" (UID: \"33d1f453-d599-4b5a-8701-76670832838e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2"
Oct 30 23:34:12.632255 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.632188    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5d4l\" (UniqueName: \"kubernetes.io/projected/33d1f453-d599-4b5a-8701-76670832838e-kube-api-access-f5d4l\") pod \"maas-api-revoke-sub-37084415-f9px2\" (UID: \"33d1f453-d599-4b5a-8701-76670832838e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2"
Oct 30 23:34:12.632859 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.632838    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33d1f453-d599-4b5a-8701-76670832838e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37084415-f9px2\" (UID: \"33d1f453-d599-4b5a-8701-76670832838e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2"
Oct 30 23:34:12.634543 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.634522    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33d1f453-d599-4b5a-8701-76670832838e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37084415-f9px2\" (UID: \"33d1f453-d599-4b5a-8701-76670832838e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2"
Oct 30 23:34:12.689739 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.689717    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5d4l\" (UniqueName: \"kubernetes.io/projected/33d1f453-d599-4b5a-8701-76670832838e-kube-api-access-f5d4l\") pod \"maas-api-revoke-sub-37084415-f9px2\" (UID: \"33d1f453-d599-4b5a-8701-76670832838e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2"
Oct 30 23:34:12.780858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:12.780825    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2"
Oct 30 23:34:13.167013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:13.166980    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2"]
Oct 30 23:34:13.168126 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:34:13.168094    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d1f453_d599_4b5a_8701_76670832838e.slice/crio-35db73314e6ae94bc9e97452c7e6f2a50ca88fa8e69586be6ee4744adda4fcb2 WatchSource:0}: Error finding container 35db73314e6ae94bc9e97452c7e6f2a50ca88fa8e69586be6ee4744adda4fcb2: Status 404 returned error can't find the container with id 35db73314e6ae94bc9e97452c7e6f2a50ca88fa8e69586be6ee4744adda4fcb2
Oct 30 23:34:14.107451 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:14.107418    5078 generic.go:356] "Generic (PLEG): container finished" podID="33d1f453-d599-4b5a-8701-76670832838e" containerID="35c5213216ed5f0548b4747bf4fde695499bfe2571090d6b09816c8108dc9a82" exitCode=0
Oct 30 23:34:14.107684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:14.107469    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2" event={"ID":"33d1f453-d599-4b5a-8701-76670832838e","Type":"ContainerDied","Data":"35c5213216ed5f0548b4747bf4fde695499bfe2571090d6b09816c8108dc9a82"}
Oct 30 23:34:14.107684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:14.107530    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2" event={"ID":"33d1f453-d599-4b5a-8701-76670832838e","Type":"ContainerStarted","Data":"35db73314e6ae94bc9e97452c7e6f2a50ca88fa8e69586be6ee4744adda4fcb2"}
Oct 30 23:34:15.300282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:15.300257    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2"
Oct 30 23:34:15.396182 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:15.396095    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33d1f453-d599-4b5a-8701-76670832838e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33d1f453-d599-4b5a-8701-76670832838e-maas-api-xks-ca\") pod \"33d1f453-d599-4b5a-8701-76670832838e\" (UID: \"33d1f453-d599-4b5a-8701-76670832838e\") "
Oct 30 23:34:15.396345 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:15.396220    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33d1f453-d599-4b5a-8701-76670832838e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33d1f453-d599-4b5a-8701-76670832838e-maas-api-openshift-ca\") pod \"33d1f453-d599-4b5a-8701-76670832838e\" (UID: \"33d1f453-d599-4b5a-8701-76670832838e\") "
Oct 30 23:34:15.396345 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:15.396276    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33d1f453-d599-4b5a-8701-76670832838e-kube-api-access-f5d4l\" (UniqueName: \"kubernetes.io/projected/33d1f453-d599-4b5a-8701-76670832838e-kube-api-access-f5d4l\") pod \"33d1f453-d599-4b5a-8701-76670832838e\" (UID: \"33d1f453-d599-4b5a-8701-76670832838e\") "
Oct 30 23:34:15.396587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:15.396565    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d1f453-d599-4b5a-8701-76670832838e-maas-api-openshift-ca" pod "33d1f453-d599-4b5a-8701-76670832838e" (UID: "33d1f453-d599-4b5a-8701-76670832838e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:34:15.397064 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:15.397041    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33d1f453-d599-4b5a-8701-76670832838e-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:15.398722 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:15.398701    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33d1f453-d599-4b5a-8701-76670832838e-maas-api-xks-ca" pod "33d1f453-d599-4b5a-8701-76670832838e" (UID: "33d1f453-d599-4b5a-8701-76670832838e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:34:15.399035 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:15.399013    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d1f453-d599-4b5a-8701-76670832838e-kube-api-access-f5d4l" pod "33d1f453-d599-4b5a-8701-76670832838e" (UID: "33d1f453-d599-4b5a-8701-76670832838e"). InnerVolumeSpecName "kube-api-access-f5d4l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:34:15.499275 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:15.499244    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5d4l\" (UniqueName: \"kubernetes.io/projected/33d1f453-d599-4b5a-8701-76670832838e-kube-api-access-f5d4l\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:15.499275 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:15.499277    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33d1f453-d599-4b5a-8701-76670832838e-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:16.125008 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:16.124983    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2"
Oct 30 23:34:16.125238 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:16.125016    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2" event={"ID":"33d1f453-d599-4b5a-8701-76670832838e","Type":"ContainerDied","Data":"35db73314e6ae94bc9e97452c7e6f2a50ca88fa8e69586be6ee4744adda4fcb2"}
Oct 30 23:34:16.125238 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:16.125070    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35db73314e6ae94bc9e97452c7e6f2a50ca88fa8e69586be6ee4744adda4fcb2"
Oct 30 23:34:16.126918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:16.126893    5078 generic.go:356] "Generic (PLEG): container finished" podID="0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" containerID="507a1dc6ac43513a08207acb61a5b7ac23838e3b2ffc993d75f01002fba6bce4" exitCode=0
Oct 30 23:34:16.127040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:16.126957    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm" event={"ID":"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7","Type":"ContainerDied","Data":"507a1dc6ac43513a08207acb61a5b7ac23838e3b2ffc993d75f01002fba6bce4"}
Oct 30 23:34:16.127402 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:16.127389    5078 scope.go:122] "RemoveContainer" containerID="507a1dc6ac43513a08207acb61a5b7ac23838e3b2ffc993d75f01002fba6bce4"
Oct 30 23:34:16.128882 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:16.128853    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:34:16.128986 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:16.128959    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:34:16.129050 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:16.129015    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:34:17.136737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:17.136706    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm" event={"ID":"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7","Type":"ContainerStarted","Data":"dc676a5de0132e297bd61c2b5704985d92b934eda8ed1a25a05619eb5ba0f8a1"}
Oct 30 23:34:17.609863 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:34:17.609835    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:34:17.834741 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:17.834708    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:34:17.835918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:17.835894    5078 patch_prober.go:29] interesting pod/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 23:34:17.836028 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:17.835952    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm" podUID="0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 23:34:18.781805 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:18.781767    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:34:18.837637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:18.834858    5078 patch_prober.go:29] interesting pod/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 23:34:18.837637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:18.834920    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm" podUID="0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 23:34:19.834703 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:19.834672    5078 patch_prober.go:29] interesting pod/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 23:34:19.835114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:19.834733    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm" podUID="0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 23:34:20.085187 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:20.085104    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b"]
Oct 30 23:34:20.118164 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:20.118120    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-279dfbc3-t697b"]
Oct 30 23:34:20.300068 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:20.300031    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e243a4-d9b3-48c3-a5e3-a548c2ea253d" path="/var/lib/kubelet/pods/75e243a4-d9b3-48c3-a5e3-a548c2ea253d/volumes"
Oct 30 23:34:20.836826 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:20.836790    5078 patch_prober.go:29] interesting pod/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 23:34:20.837511 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:20.837476    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm" podUID="0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 23:34:21.836018 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:21.835982    5078 patch_prober.go:29] interesting pod/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body=
Oct 30 23:34:21.836312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:21.836285    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm" podUID="0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused"
Oct 30 23:34:22.047683 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:22.047651    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942"]
Oct 30 23:34:22.099224 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:22.099154    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0340ff8-29942"]
Oct 30 23:34:22.301922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:22.301882    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a" path="/var/lib/kubelet/pods/0a1f15a7-80d1-469a-ad5d-b06fd58aaf3a/volumes"
Oct 30 23:34:22.839512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:22.839479    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:34:22.840797 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:22.840775    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:34:23.071088 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:23.071052    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww"]
Oct 30 23:34:23.076221 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:23.076198    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v"]
Oct 30 23:34:23.081795 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:23.081780    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d149c530-7pdww"]
Oct 30 23:34:23.086622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:23.086590    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-237d62a6-xkn6v"]
Oct 30 23:34:24.300510 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:24.300467    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15eb2805-3deb-43f1-bdd9-8cb3c9dc3309" path="/var/lib/kubelet/pods/15eb2805-3deb-43f1-bdd9-8cb3c9dc3309/volumes"
Oct 30 23:34:24.301198 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:24.301174    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad52a332-c204-4c8b-b4ef-e70474665391" path="/var/lib/kubelet/pods/ad52a332-c204-4c8b-b4ef-e70474665391/volumes"
Oct 30 23:34:25.826797 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:25.826766    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"]
Oct 30 23:34:25.827450 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:25.827434    5078 cpu_manager.go:414] "containerMap: removing container" podUID="33d1f453-d599-4b5a-8701-76670832838e" containerName="revoke-keys"
Oct 30 23:34:25.827513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:25.827453    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33d1f453-d599-4b5a-8701-76670832838e" containerName="revoke-keys"
Oct 30 23:34:25.827646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:25.827634    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="33d1f453-d599-4b5a-8701-76670832838e" containerName="revoke-keys"
Oct 30 23:34:25.831638 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:25.831585    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:25.884472 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:25.884426    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-fb14dc-openshift-default-dockercfg-lgdnx\""
Oct 30 23:34:25.955617 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:25.954646    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-workload-socket\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:25.955617 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:25.954775    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-credential-socket\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:25.955617 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:25.954853    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-token\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:25.955617 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:25.954891    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-workload-certs\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:25.955617 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:25.954974    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776ad186-9f08-44e9-99e6-f9bc82acc93c-istiod-ca-cert\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:25.955617 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:25.955040    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-envoy\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:25.955617 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:25.955068    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-podinfo\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:25.955617 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:25.955096    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-data\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:25.955617 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:25.955225    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27x9k\" (UniqueName: \"kubernetes.io/projected/776ad186-9f08-44e9-99e6-f9bc82acc93c-kube-api-access-27x9k\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:25.963505 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:25.963469    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"]
Oct 30 23:34:26.102712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.091673    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-workload-socket\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.102712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.091793    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-credential-socket\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.102712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.091865    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-token\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.102712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.091904    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-workload-certs\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.102712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.091983    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776ad186-9f08-44e9-99e6-f9bc82acc93c-istiod-ca-cert\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.102712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.092046    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-envoy\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.102712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.092073    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-podinfo\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.102712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.092149    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-data\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.102712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.092338    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27x9k\" (UniqueName: \"kubernetes.io/projected/776ad186-9f08-44e9-99e6-f9bc82acc93c-kube-api-access-27x9k\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.102712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.097423    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776ad186-9f08-44e9-99e6-f9bc82acc93c-istiod-ca-cert\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.102712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.097748    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-workload-socket\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.102712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.097984    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-credential-socket\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.127624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.126464    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-workload-certs\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.127624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.126753    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-data\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.376385 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.375382    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-podinfo\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.427844 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.420541    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27x9k\" (UniqueName: \"kubernetes.io/projected/776ad186-9f08-44e9-99e6-f9bc82acc93c-kube-api-access-27x9k\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.436027 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.435992    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-envoy\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.445635 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.445274    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-token\") pod \"e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") " pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.452837 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.452810    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:26.909402 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:34:26.909362    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776ad186_9f08_44e9_99e6_f9bc82acc93c.slice/crio-05b64633bc0689cc2375d6d8d8e98c7c2580d5de905b9f4a6bf21b14b796d688 WatchSource:0}: Error finding container 05b64633bc0689cc2375d6d8d8e98c7c2580d5de905b9f4a6bf21b14b796d688: Status 404 returned error can't find the container with id 05b64633bc0689cc2375d6d8d8e98c7c2580d5de905b9f4a6bf21b14b796d688
Oct 30 23:34:26.909886 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.909591    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"]
Oct 30 23:34:26.912158 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.912125    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:34:26.912254 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.912189    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:34:26.912254 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:26.912222    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:34:27.048645 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:27.048590    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m"]
Oct 30 23:34:27.054439 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:27.054408    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25027913-wm67m"]
Oct 30 23:34:27.476736 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:27.476699    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd" event={"ID":"776ad186-9f08-44e9-99e6-f9bc82acc93c","Type":"ContainerStarted","Data":"bf9135ff1b93ab4e7a8f6f0ac5ba38b4e8ae5bc4c48c83566f3ceb0b3810d72a"}
Oct 30 23:34:27.476954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:27.476765    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd" event={"ID":"776ad186-9f08-44e9-99e6-f9bc82acc93c","Type":"ContainerStarted","Data":"05b64633bc0689cc2375d6d8d8e98c7c2580d5de905b9f4a6bf21b14b796d688"}
Oct 30 23:34:27.595255 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:27.595197    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd" podStartSLOduration=2.595177729 podStartE2EDuration="2.595177729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:34:27.59475138 +0000 UTC m=+1653.482657732" watchObservedRunningTime="2026-10-30 23:34:27.595177729 +0000 UTC m=+1653.483084098"
Oct 30 23:34:28.299976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:28.299943    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f" path="/var/lib/kubelet/pods/31d4d0c1-bda3-494f-86f4-1a6f2dd4fd8f/volumes"
Oct 30 23:34:28.454817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:28.454789    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:28.460166 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:28.460139    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:28.484563 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:28.484535    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:28.485703 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:28.485684    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:34:31.054498 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:31.054461    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26"]
Oct 30 23:34:31.102949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:31.102920    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3df90abd-bkw26"]
Oct 30 23:34:31.833770 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:31.833739    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"]
Oct 30 23:34:31.837908 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:31.837892    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:31.893624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:31.890967    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1ce657-openshift-default-dockercfg-tj2zr\""
Oct 30 23:34:31.974624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:31.953770    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-envoy\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:31.975151 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:31.975117    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-token\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:31.975394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:31.975374    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-workload-socket\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:31.975546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:31.975520    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-workload-certs\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:31.975731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:31.975715    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b13c626-bbb0-4874-88b2-cecb77a61423-istiod-ca-cert\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.012621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.009712    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-credential-socket\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.158685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.144956    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-data\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.158685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.145360    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl7nt\" (UniqueName: \"kubernetes.io/projected/0b13c626-bbb0-4874-88b2-cecb77a61423-kube-api-access-bl7nt\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.158685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.145632    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-podinfo\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.178413 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.178380    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77"]
Oct 30 23:34:32.232075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.232016    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77"
Oct 30 23:34:32.265629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.247388    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-envoy\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.329846 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.329797    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-token\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.447941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.292990    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:34:32.448135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.293756    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:34:32.448229 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.293836    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:34:32.448311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.340351    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:34:32.449687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.277773    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"]
Oct 30 23:34:32.449687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.447901    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-workload-socket\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.449687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.448863    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-workload-certs\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.449687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.448939    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b13c626-bbb0-4874-88b2-cecb77a61423-istiod-ca-cert\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.449687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.449037    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcfdb\" (UniqueName: \"kubernetes.io/projected/0cb6378e-0174-414d-9f11-710c82c3704a-kube-api-access-rcfdb\") pod \"maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77\" (UID: \"0cb6378e-0174-414d-9f11-710c82c3704a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77"
Oct 30 23:34:32.449687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.449084    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb6378e-0174-414d-9f11-710c82c3704a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77\" (UID: \"0cb6378e-0174-414d-9f11-710c82c3704a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77"
Oct 30 23:34:32.449687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.449145    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-credential-socket\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.449687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.449169    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-data\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.449687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.449248    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb6378e-0174-414d-9f11-710c82c3704a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77\" (UID: \"0cb6378e-0174-414d-9f11-710c82c3704a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77"
Oct 30 23:34:32.449687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.449279    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl7nt\" (UniqueName: \"kubernetes.io/projected/0b13c626-bbb0-4874-88b2-cecb77a61423-kube-api-access-bl7nt\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.449687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.449417    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-podinfo\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.451425 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.448298    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-workload-socket\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.451745 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.451721    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-workload-certs\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.452362 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.452316    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b13c626-bbb0-4874-88b2-cecb77a61423-istiod-ca-cert\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.457273 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.457243    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-data\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.457678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.457661    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-credential-socket\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.464929 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.464908    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72cedd5d-767a-4e31-855f-0578981b8c3b" path="/var/lib/kubelet/pods/72cedd5d-767a-4e31-855f-0578981b8c3b/volumes"
Oct 30 23:34:32.467724 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.467705    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77"]
Oct 30 23:34:32.474512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.474485    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-podinfo\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.474938 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.474915    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-envoy\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.477305 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.477284    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-token\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.516912 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.516884    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl7nt\" (UniqueName: \"kubernetes.io/projected/0b13c626-bbb0-4874-88b2-cecb77a61423-kube-api-access-bl7nt\") pod \"e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") " pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.551070 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.551043    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcfdb\" (UniqueName: \"kubernetes.io/projected/0cb6378e-0174-414d-9f11-710c82c3704a-kube-api-access-rcfdb\") pod \"maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77\" (UID: \"0cb6378e-0174-414d-9f11-710c82c3704a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77"
Oct 30 23:34:32.551210 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.551084    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb6378e-0174-414d-9f11-710c82c3704a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77\" (UID: \"0cb6378e-0174-414d-9f11-710c82c3704a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77"
Oct 30 23:34:32.551210 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.551150    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb6378e-0174-414d-9f11-710c82c3704a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77\" (UID: \"0cb6378e-0174-414d-9f11-710c82c3704a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77"
Oct 30 23:34:32.551926 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.551899    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb6378e-0174-414d-9f11-710c82c3704a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77\" (UID: \"0cb6378e-0174-414d-9f11-710c82c3704a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77"
Oct 30 23:34:32.553508 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.553486    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb6378e-0174-414d-9f11-710c82c3704a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77\" (UID: \"0cb6378e-0174-414d-9f11-710c82c3704a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77"
Oct 30 23:34:32.609699 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.609676    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcfdb\" (UniqueName: \"kubernetes.io/projected/0cb6378e-0174-414d-9f11-710c82c3704a-kube-api-access-rcfdb\") pod \"maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77\" (UID: \"0cb6378e-0174-414d-9f11-710c82c3704a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77"
Oct 30 23:34:32.757397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.756234    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:32.813111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:32.812868    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77"
Oct 30 23:34:33.280842 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:33.280812    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77"]
Oct 30 23:34:33.281183 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:34:33.280997    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb6378e_0174_414d_9f11_710c82c3704a.slice/crio-ec5ac721a2b1d2433169a0c9078ba9fd04687f37241e2f0f860c37e0e9146832 WatchSource:0}: Error finding container ec5ac721a2b1d2433169a0c9078ba9fd04687f37241e2f0f860c37e0e9146832: Status 404 returned error can't find the container with id ec5ac721a2b1d2433169a0c9078ba9fd04687f37241e2f0f860c37e0e9146832
Oct 30 23:34:33.320823 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:34:33.320795    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b13c626_bbb0_4874_88b2_cecb77a61423.slice/crio-1e1d8dac0652ce2017820fc7d7d427604ec42ca06d01d130584c67365a35d8ba WatchSource:0}: Error finding container 1e1d8dac0652ce2017820fc7d7d427604ec42ca06d01d130584c67365a35d8ba: Status 404 returned error can't find the container with id 1e1d8dac0652ce2017820fc7d7d427604ec42ca06d01d130584c67365a35d8ba
Oct 30 23:34:33.323473 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:33.323443    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:34:33.323560 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:33.323511    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:34:33.323560 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:33.323546    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:34:33.334537 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:33.334512    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"]
Oct 30 23:34:33.525963 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:33.525929    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77" event={"ID":"0cb6378e-0174-414d-9f11-710c82c3704a","Type":"ContainerStarted","Data":"ec5ac721a2b1d2433169a0c9078ba9fd04687f37241e2f0f860c37e0e9146832"}
Oct 30 23:34:33.528363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:33.528233    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz" event={"ID":"0b13c626-bbb0-4874-88b2-cecb77a61423","Type":"ContainerStarted","Data":"e813ebaa8a7b9cdc591c908ab710b4afc85a904f813c38e3c7d98a99eb3079b9"}
Oct 30 23:34:33.528363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:33.528307    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz" event={"ID":"0b13c626-bbb0-4874-88b2-cecb77a61423","Type":"ContainerStarted","Data":"1e1d8dac0652ce2017820fc7d7d427604ec42ca06d01d130584c67365a35d8ba"}
Oct 30 23:34:33.648145 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:33.648098    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz" podStartSLOduration=2.648082521 podStartE2EDuration="2.648082521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:34:33.644817191 +0000 UTC m=+1659.532723540" watchObservedRunningTime="2026-10-30 23:34:33.648082521 +0000 UTC m=+1659.535988873"
Oct 30 23:34:33.834145 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:33.826999    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:34.539200 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:34.539166    5078 generic.go:356] "Generic (PLEG): container finished" podID="0cb6378e-0174-414d-9f11-710c82c3704a" containerID="1b2addfc5cc4a1f87a54d7e79a81c8fd4209b8539a4283b8fc32b9bdf8532655" exitCode=0
Oct 30 23:34:34.539618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:34.539241    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77" event={"ID":"0cb6378e-0174-414d-9f11-710c82c3704a","Type":"ContainerDied","Data":"1b2addfc5cc4a1f87a54d7e79a81c8fd4209b8539a4283b8fc32b9bdf8532655"}
Oct 30 23:34:34.820679 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:34.819638    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:35.547356 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:35.547330    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:35.548421 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:35.548398    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:34:35.719951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:35.719918    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77"
Oct 30 23:34:35.792167 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:35.792135    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cb6378e-0174-414d-9f11-710c82c3704a-kube-api-access-rcfdb\" (UniqueName: \"kubernetes.io/projected/0cb6378e-0174-414d-9f11-710c82c3704a-kube-api-access-rcfdb\") pod \"0cb6378e-0174-414d-9f11-710c82c3704a\" (UID: \"0cb6378e-0174-414d-9f11-710c82c3704a\") "
Oct 30 23:34:35.792355 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:35.792305    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cb6378e-0174-414d-9f11-710c82c3704a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb6378e-0174-414d-9f11-710c82c3704a-maas-api-xks-ca\") pod \"0cb6378e-0174-414d-9f11-710c82c3704a\" (UID: \"0cb6378e-0174-414d-9f11-710c82c3704a\") "
Oct 30 23:34:35.792447 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:35.792431    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cb6378e-0174-414d-9f11-710c82c3704a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb6378e-0174-414d-9f11-710c82c3704a-maas-api-openshift-ca\") pod \"0cb6378e-0174-414d-9f11-710c82c3704a\" (UID: \"0cb6378e-0174-414d-9f11-710c82c3704a\") "
Oct 30 23:34:35.792783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:35.792760    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cb6378e-0174-414d-9f11-710c82c3704a-maas-api-openshift-ca" pod "0cb6378e-0174-414d-9f11-710c82c3704a" (UID: "0cb6378e-0174-414d-9f11-710c82c3704a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:34:35.793125 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:35.793108    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cb6378e-0174-414d-9f11-710c82c3704a-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:35.794336 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:35.794316    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cb6378e-0174-414d-9f11-710c82c3704a-kube-api-access-rcfdb" pod "0cb6378e-0174-414d-9f11-710c82c3704a" (UID: "0cb6378e-0174-414d-9f11-710c82c3704a"). InnerVolumeSpecName "kube-api-access-rcfdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:34:35.794382 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:35.794354    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cb6378e-0174-414d-9f11-710c82c3704a-maas-api-xks-ca" pod "0cb6378e-0174-414d-9f11-710c82c3704a" (UID: "0cb6378e-0174-414d-9f11-710c82c3704a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:34:35.895225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:35.895191    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcfdb\" (UniqueName: \"kubernetes.io/projected/0cb6378e-0174-414d-9f11-710c82c3704a-kube-api-access-rcfdb\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:35.895225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:35.895222    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cb6378e-0174-414d-9f11-710c82c3704a-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:36.556468 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:36.556439    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77"
Oct 30 23:34:36.556468 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:36.556452    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77" event={"ID":"0cb6378e-0174-414d-9f11-710c82c3704a","Type":"ContainerDied","Data":"ec5ac721a2b1d2433169a0c9078ba9fd04687f37241e2f0f860c37e0e9146832"}
Oct 30 23:34:36.556971 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:36.556510    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec5ac721a2b1d2433169a0c9078ba9fd04687f37241e2f0f860c37e0e9146832"
Oct 30 23:34:38.058096 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:38.058064    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx"]
Oct 30 23:34:38.107527 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:38.107477    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29ec09e1-96mfx"]
Oct 30 23:34:38.299984 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:38.299954    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ab1f8c9-cc86-4e48-b679-d38e1e6ad418" path="/var/lib/kubelet/pods/3ab1f8c9-cc86-4e48-b679-d38e1e6ad418/volumes"
Oct 30 23:34:41.822633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:41.820691    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:42.082680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:42.066688    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:42.082680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:42.067088    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded"
Oct 30 23:34:42.082680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:42.072669    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:42.225624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:42.224675    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 23:34:42.225624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:42.225528    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:42.225895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:42.225817    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 23:34:42.229619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:42.226827    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 23:34:42.352470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:42.349680    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:42.352470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:42.350501    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:42.352470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:42.351172    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:42.352470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:42.351412    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 23:34:42.444097 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:42.443993    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:42.607507 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:42.606682    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:42.829333 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:42.822231    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:42.841616 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:42.840068    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:34:42.885803 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:42.885722    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr"]
Oct 30 23:34:42.980918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:42.980873    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:43.274221 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.274126    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d"]
Oct 30 23:34:43.294646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.276724    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:43.294646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.276854    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 23:34:43.294646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.276898    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 23:34:43.294646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.276940    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 23:34:43.294646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.277105    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:43.294646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.277143    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 23:34:43.294646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.277176    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:43.350331 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.350284    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:43.351696 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.351645    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:43.594623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.351901    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 23:34:43.594623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.351444    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:43.624677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.542197    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:34:43.624677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.621970    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:34:43.624677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.622040    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:34:43.624677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.622082    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:34:43.624677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.622122    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:34:43.624677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.622159    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:34:43.624677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.622194    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:34:43.624677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.622226    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:34:43.624677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.622261    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:34:43.624677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.622294    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:34:43.624677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.622332    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:34:43.639765 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.636828    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:43.639765 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.638056    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:34:43.730624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.566526    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:43.874881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:43.874792    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:44.025623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:44.014434    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:34:44.045307 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:44.035648    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:44.513725 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:44.510076    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:44.513725 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:44.510364    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 23:34:44.513725 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:44.510549    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 23:34:44.513725 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:44.510746    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 23:34:44.513725 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:44.511288    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:44.516713 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:44.516683    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:44.551938 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:44.551893    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 23:34:44.597998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:44.587667    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 23:34:44.597998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:44.590773    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:44.597998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:44.597768    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:44.600035 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:44.600000    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:44.728173 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:44.694728    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Oct 30 23:34:44.937636 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:44.879717    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:45.063960 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:45.063910    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:45.393063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:45.393005    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:45.515961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:45.515903    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:45.527709 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:45.518735    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:45.542637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:45.518801    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 23:34:45.542637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:45.518833    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:45.542637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:45.518881    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded"
Oct 30 23:34:45.607130 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:45.607085    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:45.629052 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:45.628996    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:45.638625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:45.635220    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 23:34:45.663690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:45.651713    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 23:34:45.710577 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:45.710521    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Oct 30 23:34:45.784760 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:45.784716    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:45.878426 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:45.878371    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:45.899723 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:45.899686    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/health\": context deadline exceeded"
Oct 30 23:34:46.149624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:46.108314    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 23:34:46.310667 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:46.310638    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:34:46.318627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:46.181336    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 30 23:34:46.553485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:46.553441    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 23:34:46.906690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:46.881986    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": context deadline exceeded"
Oct 30 23:34:47.071647 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:46.919096    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded"
Oct 30 23:34:47.071647 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:46.919399    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:47.096783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:46.919701    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Oct 30 23:34:47.122620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:46.947503    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:47.122620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:46.947675    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 23:34:47.123762 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:46.948841    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:47.210009 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:46.952849    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded"
Oct 30 23:34:47.341521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:47.341477    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:47.341924 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:47.341895    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded"
Oct 30 23:34:47.386623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:47.379023    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:47.873417 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:47.855800    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 23:34:48.013271 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:47.949913    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-rwc77 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:34:48.013271 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:47.949995    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77" podUID="f434e905-62f0-4aef-acf5-1ba6f49ff6f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.25:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:48.071926 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.071883    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:48.072313 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.071857    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:48.080756 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.080711    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:48.105887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.105821    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:48.106478 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.106425    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:48.106856 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.106832    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded"
Oct 30 23:34:48.160324 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.160217    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded"
Oct 30 23:34:48.161879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.161834    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 23:34:48.163518 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.163497    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 23:34:48.214997 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.211995    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr" podUID="1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b" containerName="payload-processing" containerID="cri-o://8a8d620328edf96328187e5d246a3e0658b99efa8aa634488e84de68d870ba2b" gracePeriod=25
Oct 30 23:34:48.236277 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.236219    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 30 23:34:48.342622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.325645    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:48.567660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.343689    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8000/ready\": context deadline exceeded"
Oct 30 23:34:48.567660 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:34:48.547258    5078 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.205s"
Oct 30 23:34:48.591753 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.591573    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.82:8000/ready\": context deadline exceeded"
Oct 30 23:34:48.619653 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.570266    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d" podUID="4386f5ab-5a83-4be1-af56-255e4ee9b5da" containerName="payload-pre-processing" containerID="cri-o://c69ed29a03136a5424d5394ef0bce65767931efec76d4685dd8b7d8ae8e1e7a9" gracePeriod=25
Oct 30 23:34:48.619894 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:34:48.591762    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:34:48.649550 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.649514    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:48.762181 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.762154    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:34:48.762465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.762449    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:34:48.762631 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.762617    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:34:48.763015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.762999    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:34:48.763170 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.763159    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:34:48.773040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.773022    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn"
Oct 30 23:34:48.854834 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.838607    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:34:48.856129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.856048    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:34:48.918352 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.918227    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:34:48.941140 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.940695    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:34:48.947302 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:48.942245    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:34:49.041984 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.041949    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:34:49.045510 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.045220    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:34:49.048403 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.047693    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:34:49.054069 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.054048    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp"]
Oct 30 23:34:49.124293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.124203    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-666c817c-jh6hp"]
Oct 30 23:34:49.565551 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.565396    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr"
Oct 30 23:34:49.627365 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.627335    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b-kube-api-access-7gq5j\" (UniqueName: \"kubernetes.io/projected/1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b-kube-api-access-7gq5j\") pod \"1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b\" (UID: \"1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b\") "
Oct 30 23:34:49.627536 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.627501    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b-plugins-config-volume\") pod \"1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b\" (UID: \"1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b\") "
Oct 30 23:34:49.628578 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.628552    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b-plugins-config-volume" pod "1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b" (UID: "1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:34:49.633905 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.633501    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b-kube-api-access-7gq5j" pod "1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b" (UID: "1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b"). InnerVolumeSpecName "kube-api-access-7gq5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:34:49.643625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.643565    5078 generic.go:356] "Generic (PLEG): container finished" podID="1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b" containerID="8a8d620328edf96328187e5d246a3e0658b99efa8aa634488e84de68d870ba2b" exitCode=0
Oct 30 23:34:49.643763 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.643734    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr" event={"ID":"1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b","Type":"ContainerDied","Data":"8a8d620328edf96328187e5d246a3e0658b99efa8aa634488e84de68d870ba2b"}
Oct 30 23:34:49.643828 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.643786    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr" event={"ID":"1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b","Type":"ContainerDied","Data":"9833b91d6ad1d2cb8e8357575405a3ea904a3080ba81523dbeab3d7410adae24"}
Oct 30 23:34:49.643828 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.643818    5078 scope.go:122] "RemoveContainer" containerID="8a8d620328edf96328187e5d246a3e0658b99efa8aa634488e84de68d870ba2b"
Oct 30 23:34:49.643948 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.643884    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr"
Oct 30 23:34:49.648563 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.648541    5078 generic.go:356] "Generic (PLEG): container finished" podID="4386f5ab-5a83-4be1-af56-255e4ee9b5da" containerID="c69ed29a03136a5424d5394ef0bce65767931efec76d4685dd8b7d8ae8e1e7a9" exitCode=0
Oct 30 23:34:49.649066 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.649045    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d" event={"ID":"4386f5ab-5a83-4be1-af56-255e4ee9b5da","Type":"ContainerDied","Data":"c69ed29a03136a5424d5394ef0bce65767931efec76d4685dd8b7d8ae8e1e7a9"}
Oct 30 23:34:49.649230 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.649214    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d" event={"ID":"4386f5ab-5a83-4be1-af56-255e4ee9b5da","Type":"ContainerDied","Data":"c776ab6457270c83b111e6a08baa1413e0a5a141d8bd4ff484430740e7fe0ad3"}
Oct 30 23:34:49.649350 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.649337    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c776ab6457270c83b111e6a08baa1413e0a5a141d8bd4ff484430740e7fe0ad3"
Oct 30 23:34:49.691142 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.690913    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d"
Oct 30 23:34:49.705281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.705194    5078 scope.go:122] "RemoveContainer" containerID="8a8d620328edf96328187e5d246a3e0658b99efa8aa634488e84de68d870ba2b"
Oct 30 23:34:49.706960 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:34:49.706908    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a8d620328edf96328187e5d246a3e0658b99efa8aa634488e84de68d870ba2b\": container with ID starting with 8a8d620328edf96328187e5d246a3e0658b99efa8aa634488e84de68d870ba2b not found: ID does not exist" containerID="8a8d620328edf96328187e5d246a3e0658b99efa8aa634488e84de68d870ba2b"
Oct 30 23:34:49.707090 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.706952    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a8d620328edf96328187e5d246a3e0658b99efa8aa634488e84de68d870ba2b"} err="failed to get container status \"8a8d620328edf96328187e5d246a3e0658b99efa8aa634488e84de68d870ba2b\": rpc error: code = NotFound desc = could not find container \"8a8d620328edf96328187e5d246a3e0658b99efa8aa634488e84de68d870ba2b\": container with ID starting with 8a8d620328edf96328187e5d246a3e0658b99efa8aa634488e84de68d870ba2b not found: ID does not exist"
Oct 30 23:34:49.728566 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.728540    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4386f5ab-5a83-4be1-af56-255e4ee9b5da-kube-api-access-jwvx8\" (UniqueName: \"kubernetes.io/projected/4386f5ab-5a83-4be1-af56-255e4ee9b5da-kube-api-access-jwvx8\") pod \"4386f5ab-5a83-4be1-af56-255e4ee9b5da\" (UID: \"4386f5ab-5a83-4be1-af56-255e4ee9b5da\") "
Oct 30 23:34:49.729055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.729036    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4386f5ab-5a83-4be1-af56-255e4ee9b5da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4386f5ab-5a83-4be1-af56-255e4ee9b5da-plugins-config-volume\") pod \"4386f5ab-5a83-4be1-af56-255e4ee9b5da\" (UID: \"4386f5ab-5a83-4be1-af56-255e4ee9b5da\") "
Oct 30 23:34:49.729457 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.729429    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4386f5ab-5a83-4be1-af56-255e4ee9b5da-plugins-config-volume" pod "4386f5ab-5a83-4be1-af56-255e4ee9b5da" (UID: "4386f5ab-5a83-4be1-af56-255e4ee9b5da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:34:49.729919 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.729899    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4386f5ab-5a83-4be1-af56-255e4ee9b5da-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:49.730045 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.730032    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gq5j\" (UniqueName: \"kubernetes.io/projected/1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b-kube-api-access-7gq5j\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:49.730132 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.730122    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:49.731791 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.731767    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4386f5ab-5a83-4be1-af56-255e4ee9b5da-kube-api-access-jwvx8" pod "4386f5ab-5a83-4be1-af56-255e4ee9b5da" (UID: "4386f5ab-5a83-4be1-af56-255e4ee9b5da"). InnerVolumeSpecName "kube-api-access-jwvx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:34:49.837618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:49.834570    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwvx8\" (UniqueName: \"kubernetes.io/projected/4386f5ab-5a83-4be1-af56-255e4ee9b5da-kube-api-access-jwvx8\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:34:50.300546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:50.300514    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="528ef440-0f7c-4c42-b90f-55721ebf655b" path="/var/lib/kubelet/pods/528ef440-0f7c-4c42-b90f-55721ebf655b/volumes"
Oct 30 23:34:50.662878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:50.662789    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d"
Oct 30 23:34:51.010163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:51.010090    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr"]
Oct 30 23:34:51.019480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:51.019441    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-20f33adb-644f7dc467-w96cr"]
Oct 30 23:34:51.195620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:51.195577    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d"]
Oct 30 23:34:51.199267 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:51.199239    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-20f33adb-77d5d677c5-9kd9d"]
Oct 30 23:34:52.301307 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:52.301268    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b" path="/var/lib/kubelet/pods/1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b/volumes"
Oct 30 23:34:52.301916 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:52.301900    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4386f5ab-5a83-4be1-af56-255e4ee9b5da" path="/var/lib/kubelet/pods/4386f5ab-5a83-4be1-af56-255e4ee9b5da/volumes"
Oct 30 23:34:53.635870 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.635836    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"]
Oct 30 23:34:53.636629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.636584    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b" containerName="payload-processing"
Oct 30 23:34:53.636629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.636617    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b" containerName="payload-processing"
Oct 30 23:34:53.636817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.636657    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4386f5ab-5a83-4be1-af56-255e4ee9b5da" containerName="payload-pre-processing"
Oct 30 23:34:53.636817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.636663    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4386f5ab-5a83-4be1-af56-255e4ee9b5da" containerName="payload-pre-processing"
Oct 30 23:34:53.636817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.636686    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0cb6378e-0174-414d-9f11-710c82c3704a" containerName="revoke-keys"
Oct 30 23:34:53.636817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.636692    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cb6378e-0174-414d-9f11-710c82c3704a" containerName="revoke-keys"
Oct 30 23:34:53.636817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.636775    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4386f5ab-5a83-4be1-af56-255e4ee9b5da" containerName="payload-pre-processing"
Oct 30 23:34:53.636817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.636786    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cb6378e-0174-414d-9f11-710c82c3704a" containerName="revoke-keys"
Oct 30 23:34:53.636817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.636794    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf17ea6-8cc2-4a4b-9e1f-ceeacc389f9b" containerName="payload-processing"
Oct 30 23:34:53.666996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.666969    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:53.773379 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.773347    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"]
Oct 30 23:34:53.841407 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.841370    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc8pt\" (UniqueName: \"kubernetes.io/projected/7a62b6f7-1528-4540-8793-c333094fb427-kube-api-access-gc8pt\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.841489    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.841588    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a62b6f7-1528-4540-8793-c333094fb427-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.841696    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.841756    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.841778    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a62b6f7-1528-4540-8793-c333094fb427-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.841924    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a62b6f7-1528-4540-8793-c333094fb427-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.841949    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.841965    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.943712    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc8pt\" (UniqueName: \"kubernetes.io/projected/7a62b6f7-1528-4540-8793-c333094fb427-kube-api-access-gc8pt\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.943801    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.943869    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a62b6f7-1528-4540-8793-c333094fb427-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.943977    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.944081    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.944121    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a62b6f7-1528-4540-8793-c333094fb427-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.944196    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a62b6f7-1528-4540-8793-c333094fb427-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.944224    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.944232    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.944277    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.944327    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.944432    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.944539    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.944566    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a62b6f7-1528-4540-8793-c333094fb427-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.946632    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:53.946773    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a62b6f7-1528-4540-8793-c333094fb427-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:54.003046    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a62b6f7-1528-4540-8793-c333094fb427-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:54.003365    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc8pt\" (UniqueName: \"kubernetes.io/projected/7a62b6f7-1528-4540-8793-c333094fb427-kube-api-access-gc8pt\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-mj8mz\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.430855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:54.297457    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:54.750405 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:54.750381    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"]
Oct 30 23:34:54.794433 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:34:54.794405    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a62b6f7_1528_4540_8793_c333094fb427.slice/crio-a1bae3f1e1e2fbbc63c461e11744d932f35fb447485362bf19e95e7cd099c270 WatchSource:0}: Error finding container a1bae3f1e1e2fbbc63c461e11744d932f35fb447485362bf19e95e7cd099c270: Status 404 returned error can't find the container with id a1bae3f1e1e2fbbc63c461e11744d932f35fb447485362bf19e95e7cd099c270
Oct 30 23:34:54.797237 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:54.797208    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:34:54.797315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:54.797296    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:34:54.797354 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:54.797326    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:34:55.754337 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:55.754300    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz" event={"ID":"7a62b6f7-1528-4540-8793-c333094fb427","Type":"ContainerStarted","Data":"9254b9beda66889b5d4685f847cc27e6d808560898e3f85a639bd8dc27f5561f"}
Oct 30 23:34:55.755082 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:55.755054    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz" event={"ID":"7a62b6f7-1528-4540-8793-c333094fb427","Type":"ContainerStarted","Data":"a1bae3f1e1e2fbbc63c461e11744d932f35fb447485362bf19e95e7cd099c270"}
Oct 30 23:34:55.761690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:55.760160    5078 generic.go:356] "Generic (PLEG): container finished" podID="fc173733-80d8-4793-b1a1-723844b99673" containerID="a9a1da734944bbd0a55c88375283d87a7fc4634a4df94df2bd704ab535c62b27" exitCode=0
Oct 30 23:34:55.761690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:55.760438    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r" event={"ID":"fc173733-80d8-4793-b1a1-723844b99673","Type":"ContainerDied","Data":"a9a1da734944bbd0a55c88375283d87a7fc4634a4df94df2bd704ab535c62b27"}
Oct 30 23:34:55.761690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:55.761046    5078 scope.go:122] "RemoveContainer" containerID="a9a1da734944bbd0a55c88375283d87a7fc4634a4df94df2bd704ab535c62b27"
Oct 30 23:34:55.764750 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:55.763990    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:34:55.764750 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:55.764084    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:34:55.764750 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:55.764134    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:34:55.837674 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:55.836192    5078 generic.go:356] "Generic (PLEG): container finished" podID="dc5a135a-f306-48a0-929e-3d9e44f97e1f" containerID="a97256900f72473dae2680bdd03f16d45793d684590c300bfe40fe3927a38e46" exitCode=0
Oct 30 23:34:55.837674 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:55.836243    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn" event={"ID":"dc5a135a-f306-48a0-929e-3d9e44f97e1f","Type":"ContainerDied","Data":"a97256900f72473dae2680bdd03f16d45793d684590c300bfe40fe3927a38e46"}
Oct 30 23:34:55.837674 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:55.837077    5078 scope.go:122] "RemoveContainer" containerID="a97256900f72473dae2680bdd03f16d45793d684590c300bfe40fe3927a38e46"
Oct 30 23:34:55.846284 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:55.846252    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:34:55.846492 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:55.846465    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:34:55.846801 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:55.846586    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:34:56.105961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:56.105685    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz" podStartSLOduration=3.105664099 podStartE2EDuration="3.105664099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:34:55.986906639 +0000 UTC m=+1681.874812995" watchObservedRunningTime="2026-10-30 23:34:56.105664099 +0000 UTC m=+1681.993570452"
Oct 30 23:34:56.267621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:56.261691    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:34:56.299163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:56.299128    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 23:34:56.299406 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:56.299379    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz" podUID="7a62b6f7-1528-4540-8793-c333094fb427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 23:34:56.315640 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:56.314758    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:34:56.847061 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:56.847023    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r" event={"ID":"fc173733-80d8-4793-b1a1-723844b99673","Type":"ContainerStarted","Data":"4149f24d3cc318e962f2c40a15f315175b0a27b626727cfc1f3571cd2f6f0fad"}
Oct 30 23:34:56.850879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:56.850853    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn" event={"ID":"dc5a135a-f306-48a0-929e-3d9e44f97e1f","Type":"ContainerStarted","Data":"b45665448e67bd7b57a37bb5c96d6b897366b977a22fdfdbd41f87ca0c6d62dd"}
Oct 30 23:34:56.945587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:56.945554    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:34:56.946762 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:56.946738    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 23:34:56.946887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:56.946793    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r" podUID="fc173733-80d8-4793-b1a1-723844b99673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 23:34:57.311626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:57.301015    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 23:34:57.311626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:57.301077    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz" podUID="7a62b6f7-1528-4540-8793-c333094fb427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 23:34:57.435625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:57.434578    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:34:57.435625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:57.434753    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-77jfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 23:34:57.435625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:57.434802    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn" podUID="dc5a135a-f306-48a0-929e-3d9e44f97e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 23:34:57.945087 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:57.945061    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 23:34:57.945556 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:57.945534    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r" podUID="fc173733-80d8-4793-b1a1-723844b99673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 23:34:58.313622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:58.309246    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 23:34:58.313622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:58.309303    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz" podUID="7a62b6f7-1528-4540-8793-c333094fb427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 23:34:58.434021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:58.433985    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-77jfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 23:34:58.434325 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:58.434295    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn" podUID="dc5a135a-f306-48a0-929e-3d9e44f97e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 23:34:58.945435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:58.945390    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 23:34:58.945875 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:58.945457    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r" podUID="fc173733-80d8-4793-b1a1-723844b99673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 23:34:59.330623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:59.329730    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 23:34:59.330623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:59.329796    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz" podUID="7a62b6f7-1528-4540-8793-c333094fb427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 23:34:59.438193 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:59.438080    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-77jfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 23:34:59.438193 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:59.438137    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn" podUID="dc5a135a-f306-48a0-929e-3d9e44f97e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 23:34:59.945140 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:59.945102    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 23:34:59.945349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:34:59.945152    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r" podUID="fc173733-80d8-4793-b1a1-723844b99673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 23:35:00.319623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:00.318870    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body=
Oct 30 23:35:00.319623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:00.318928    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz" podUID="7a62b6f7-1528-4540-8793-c333094fb427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused"
Oct 30 23:35:00.335429 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:00.335397    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:35:00.437316 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:00.437281    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-77jfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 23:35:00.437643 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:00.437613    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn" podUID="dc5a135a-f306-48a0-929e-3d9e44f97e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 23:35:00.823308 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:00.823274    5078 scope.go:122] "RemoveContainer" containerID="963e4190b22bed3990e92e28c50e46cb39000035f9d6368821bcc2efc71cb10f"
Oct 30 23:35:00.843780 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:00.843755    5078 scope.go:122] "RemoveContainer" containerID="24c856a20e568c3eff2bebe8765c5c38fe2e0cc431e9229ba4050c5dda957d35"
Oct 30 23:35:00.854419 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:00.854385    5078 scope.go:122] "RemoveContainer" containerID="6d3f35723c3a0b747d1ec5b5bc479f14488373f2aed275300ec24bdaefb5777e"
Oct 30 23:35:00.926522 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:00.926491    5078 scope.go:122] "RemoveContainer" containerID="a05bbd305c6199488691103303a37d90fd09ec04836213e3718a7b45f1cbe1d9"
Oct 30 23:35:00.942778 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:00.942756    5078 scope.go:122] "RemoveContainer" containerID="7732e8d5ff8dbcd2875e381e7f5b9239328d4fd97741496d325298b7717e1faa"
Oct 30 23:35:00.948975 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:00.948949    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 23:35:00.949327 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:00.949299    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r" podUID="fc173733-80d8-4793-b1a1-723844b99673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 23:35:01.017917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:01.017777    5078 scope.go:122] "RemoveContainer" containerID="7614ae45d3c00f9148e7699b8cddc68975d72397150717772c7229874b67ec7d"
Oct 30 23:35:01.037161 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:01.036515    5078 scope.go:122] "RemoveContainer" containerID="5c146234b9514d86a5b35b61431b884d39ae0799fe6a3db786526b41fb7ca94d"
Oct 30 23:35:01.168991 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:01.168838    5078 scope.go:122] "RemoveContainer" containerID="bef2e68cfb57c5f2cc2e0eab8cb0039eeb5e4614efe418fb7a5032780e521bde"
Oct 30 23:35:01.334930 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:01.307756    5078 scope.go:122] "RemoveContainer" containerID="3741dad639941b454b9573c1a0f43ac5afd3f226e900eff303f656f16b0fa9d4"
Oct 30 23:35:01.411621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:01.408264    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:35:01.411621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:01.410363    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:35:01.411621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:01.410904    5078 scope.go:122] "RemoveContainer" containerID="8899f2e38b7515202d2f62a76a84f78db0aa5958085b812e066b4402ef6407d8"
Oct 30 23:35:01.412932 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:01.412914    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:35:01.432345 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:01.432320    5078 scope.go:122] "RemoveContainer" containerID="67c92f7f4cd527d4a486f38041e1f9aaa44b64392972de3ae5318b20b469b3e0"
Oct 30 23:35:01.432692 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:01.432671    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-77jfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body=
Oct 30 23:35:01.432773 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:01.432719    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn" podUID="dc5a135a-f306-48a0-929e-3d9e44f97e1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused"
Oct 30 23:35:01.444565 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:01.444545    5078 scope.go:122] "RemoveContainer" containerID="cad926f3da76b32be863ace00eb465d02b876c2324525f07c4bea6cc88feac08"
Oct 30 23:35:01.945647 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:01.945529    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Oct 30 23:35:01.945871 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:01.945842    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r" podUID="fc173733-80d8-4793-b1a1-723844b99673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Oct 30 23:35:02.477700 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:02.477663    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:35:02.478960 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:02.478929    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:35:02.481728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:02.481679    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:35:02.950036 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:02.950006    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:35:02.951267 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:02.951245    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:35:04.878811 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:04.876126    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk"]
Oct 30 23:35:04.892848 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:04.890335    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk"]
Oct 30 23:35:04.892848 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:04.890403    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz"]
Oct 30 23:35:04.893441 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:04.893416    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk"
Oct 30 23:35:04.896844 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:04.896810    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz"]
Oct 30 23:35:04.897041 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:04.896914    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz"
Oct 30 23:35:04.967132 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:04.967093    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252b60c8-71fc-470b-91d3-7f6586510c69-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk\" (UID: \"252b60c8-71fc-470b-91d3-7f6586510c69\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk"
Oct 30 23:35:04.967327 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:04.967166    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7506656-8e86-4ecc-a0d7-41d6c87ddb67-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz\" (UID: \"b7506656-8e86-4ecc-a0d7-41d6c87ddb67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz"
Oct 30 23:35:04.967327 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:04.967229    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2865t\" (UniqueName: \"kubernetes.io/projected/b7506656-8e86-4ecc-a0d7-41d6c87ddb67-kube-api-access-2865t\") pod \"payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz\" (UID: \"b7506656-8e86-4ecc-a0d7-41d6c87ddb67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz"
Oct 30 23:35:04.967425 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:04.967388    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmlw6\" (UniqueName: \"kubernetes.io/projected/252b60c8-71fc-470b-91d3-7f6586510c69-kube-api-access-lmlw6\") pod \"payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk\" (UID: \"252b60c8-71fc-470b-91d3-7f6586510c69\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk"
Oct 30 23:35:04.995661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:04.995621    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-59d267-dockercfg-6kw9t\""
Oct 30 23:35:04.995661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:04.995625    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-59d267\""
Oct 30 23:35:05.068608 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.068575    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252b60c8-71fc-470b-91d3-7f6586510c69-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk\" (UID: \"252b60c8-71fc-470b-91d3-7f6586510c69\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk"
Oct 30 23:35:05.068805 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.068691    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7506656-8e86-4ecc-a0d7-41d6c87ddb67-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz\" (UID: \"b7506656-8e86-4ecc-a0d7-41d6c87ddb67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz"
Oct 30 23:35:05.068805 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.068719    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2865t\" (UniqueName: \"kubernetes.io/projected/b7506656-8e86-4ecc-a0d7-41d6c87ddb67-kube-api-access-2865t\") pod \"payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz\" (UID: \"b7506656-8e86-4ecc-a0d7-41d6c87ddb67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz"
Oct 30 23:35:05.068944 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.068923    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmlw6\" (UniqueName: \"kubernetes.io/projected/252b60c8-71fc-470b-91d3-7f6586510c69-kube-api-access-lmlw6\") pod \"payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk\" (UID: \"252b60c8-71fc-470b-91d3-7f6586510c69\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk"
Oct 30 23:35:05.069424 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.069390    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252b60c8-71fc-470b-91d3-7f6586510c69-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk\" (UID: \"252b60c8-71fc-470b-91d3-7f6586510c69\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk"
Oct 30 23:35:05.069535 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.069439    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7506656-8e86-4ecc-a0d7-41d6c87ddb67-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz\" (UID: \"b7506656-8e86-4ecc-a0d7-41d6c87ddb67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz"
Oct 30 23:35:05.128129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.128095    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmlw6\" (UniqueName: \"kubernetes.io/projected/252b60c8-71fc-470b-91d3-7f6586510c69-kube-api-access-lmlw6\") pod \"payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk\" (UID: \"252b60c8-71fc-470b-91d3-7f6586510c69\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk"
Oct 30 23:35:05.128308 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.128269    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2865t\" (UniqueName: \"kubernetes.io/projected/b7506656-8e86-4ecc-a0d7-41d6c87ddb67-kube-api-access-2865t\") pod \"payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz\" (UID: \"b7506656-8e86-4ecc-a0d7-41d6c87ddb67\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz"
Oct 30 23:35:05.209946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.209865    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk"
Oct 30 23:35:05.216166 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.216143    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz"
Oct 30 23:35:05.597897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.597864    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk"]
Oct 30 23:35:05.636040 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:35:05.636014    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252b60c8_71fc_470b_91d3_7f6586510c69.slice/crio-11918711f60f625f677d2b8fe8825fcb6d6ddc40c0596e85f26d57f1e417386f WatchSource:0}: Error finding container 11918711f60f625f677d2b8fe8825fcb6d6ddc40c0596e85f26d57f1e417386f: Status 404 returned error can't find the container with id 11918711f60f625f677d2b8fe8825fcb6d6ddc40c0596e85f26d57f1e417386f
Oct 30 23:35:05.647292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.647265    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz"]
Oct 30 23:35:05.650335 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:35:05.650307    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7506656_8e86_4ecc_a0d7_41d6c87ddb67.slice/crio-383479a2f680110a2ba81850f14693e9936d6af3485f148925e769945172883b WatchSource:0}: Error finding container 383479a2f680110a2ba81850f14693e9936d6af3485f148925e769945172883b: Status 404 returned error can't find the container with id 383479a2f680110a2ba81850f14693e9936d6af3485f148925e769945172883b
Oct 30 23:35:05.755487 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.755452    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz" event={"ID":"b7506656-8e86-4ecc-a0d7-41d6c87ddb67","Type":"ContainerStarted","Data":"28859c60576178ab9cc24ad010bb88ca64f4141257a06a740128062d662645fd"}
Oct 30 23:35:05.755704 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.755512    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz" event={"ID":"b7506656-8e86-4ecc-a0d7-41d6c87ddb67","Type":"ContainerStarted","Data":"383479a2f680110a2ba81850f14693e9936d6af3485f148925e769945172883b"}
Oct 30 23:35:05.755704 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.755546    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz"
Oct 30 23:35:05.757179 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.757158    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk" event={"ID":"252b60c8-71fc-470b-91d3-7f6586510c69","Type":"ContainerStarted","Data":"5e164c8c5e29b867e04c28493e87a8542dcad3374edb22ac1f8a1d575a5188e4"}
Oct 30 23:35:05.757287 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.757194    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk" event={"ID":"252b60c8-71fc-470b-91d3-7f6586510c69","Type":"ContainerStarted","Data":"11918711f60f625f677d2b8fe8825fcb6d6ddc40c0596e85f26d57f1e417386f"}
Oct 30 23:35:05.757287 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.757235    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk"
Oct 30 23:35:05.871048 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:05.870958    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz" podStartSLOduration=1.870944854 podStartE2EDuration="1.870944854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:35:05.86837526 +0000 UTC m=+1691.756281613" watchObservedRunningTime="2026-10-30 23:35:05.870944854 +0000 UTC m=+1691.758851206"
Oct 30 23:35:06.020558 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:06.019081    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk" podStartSLOduration=2.019057554 podStartE2EDuration="2.019057554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:35:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:35:05.991539669 +0000 UTC m=+1691.879446026" watchObservedRunningTime="2026-10-30 23:35:06.019057554 +0000 UTC m=+1691.906963908"
Oct 30 23:35:16.768304 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:16.768275    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz"
Oct 30 23:35:16.768790 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:16.768340    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk"
Oct 30 23:35:17.593871 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:35:17.593814    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:35:22.452573 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:22.452505    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77"]
Oct 30 23:35:22.500772 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:22.500715    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-20f33adb-k4l77"]
Oct 30 23:35:22.527231 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:22.527196    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"]
Oct 30 23:35:22.527501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:22.527454    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm" podUID="0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" containerName="istio-proxy" containerID="cri-o://dc676a5de0132e297bd61c2b5704985d92b934eda8ed1a25a05619eb5ba0f8a1" gracePeriod=30
Oct 30 23:35:22.919470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:22.918835    5078 patch_prober.go:29] interesting pod/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:35:22.919470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:22.918905    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm" podUID="0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:35:24.372744 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:24.372700    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cb6378e-0174-414d-9f11-710c82c3704a" path="/var/lib/kubelet/pods/0cb6378e-0174-414d-9f11-710c82c3704a/volumes"
Oct 30 23:35:27.809754 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.809725    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:35:27.930382 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.930292    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-podinfo\") pod \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") "
Oct 30 23:35:27.939617 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.930647    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-kube-api-access-7nflk\" (UniqueName: \"kubernetes.io/projected/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-kube-api-access-7nflk\") pod \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") "
Oct 30 23:35:27.939985 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.939959    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-envoy\") pod \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") "
Oct 30 23:35:27.940172 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.940157    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-credential-socket\") pod \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") "
Oct 30 23:35:27.941359 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.940389    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-token\" (UniqueName: \"kubernetes.io/projected/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-token\") pod \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") "
Oct 30 23:35:27.942458 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.942436    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-data\") pod \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") "
Oct 30 23:35:27.944166 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.944142    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istiod-ca-cert\") pod \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") "
Oct 30 23:35:27.944532 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.944504    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-workload-certs\") pod \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") "
Oct 30 23:35:27.944721 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.944545    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-workload-socket\") pod \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\" (UID: \"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7\") "
Oct 30 23:35:27.962012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.943660    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-data" pod "0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" (UID: "0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:27.962012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.945301    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-credential-socket" pod "0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" (UID: "0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:27.962012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.944945    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istiod-ca-cert" pod "0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" (UID: "0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:35:27.962012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.945524    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-workload-socket" pod "0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" (UID: "0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:27.962012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.945651    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-workload-certs" pod "0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" (UID: "0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:27.962012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.956936    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:27.962012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.956966    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:27.962012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.956982    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:27.962012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.956995    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:27.962012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.957008    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:27.962012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.959327    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-token" pod "0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" (UID: "0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:35:27.975706 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.968750    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-envoy" pod "0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" (UID: "0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:27.975706 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.971511    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-kube-api-access-7nflk" pod "0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" (UID: "0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7"). InnerVolumeSpecName "kube-api-access-7nflk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:35:27.988141 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:27.987673    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-podinfo" pod "0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" (UID: "0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:35:28.094804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:28.094740    5078 generic.go:356] "Generic (PLEG): container finished" podID="0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" containerID="dc676a5de0132e297bd61c2b5704985d92b934eda8ed1a25a05619eb5ba0f8a1" exitCode=0
Oct 30 23:35:28.095008 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:28.094882    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm" event={"ID":"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7","Type":"ContainerDied","Data":"dc676a5de0132e297bd61c2b5704985d92b934eda8ed1a25a05619eb5ba0f8a1"}
Oct 30 23:35:28.095008 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:28.094955    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm" event={"ID":"0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7","Type":"ContainerDied","Data":"0bc047a0e77981e3f9f3cf794591520e908d873e325370063df526f5055ecda7"}
Oct 30 23:35:28.095154 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:28.095103    5078 scope.go:122] "RemoveContainer" containerID="dc676a5de0132e297bd61c2b5704985d92b934eda8ed1a25a05619eb5ba0f8a1"
Oct 30 23:35:28.095277 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:28.095260    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"
Oct 30 23:35:28.100296 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:28.100261    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:28.100423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:28.100326    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nflk\" (UniqueName: \"kubernetes.io/projected/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-kube-api-access-7nflk\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:28.100423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:28.100347    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:28.100423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:28.100361    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:28.115484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:28.114899    5078 scope.go:122] "RemoveContainer" containerID="507a1dc6ac43513a08207acb61a5b7ac23838e3b2ffc993d75f01002fba6bce4"
Oct 30 23:35:28.153991 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:28.153898    5078 scope.go:122] "RemoveContainer" containerID="dc676a5de0132e297bd61c2b5704985d92b934eda8ed1a25a05619eb5ba0f8a1"
Oct 30 23:35:28.156184 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:35:28.155819    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc676a5de0132e297bd61c2b5704985d92b934eda8ed1a25a05619eb5ba0f8a1\": container with ID starting with dc676a5de0132e297bd61c2b5704985d92b934eda8ed1a25a05619eb5ba0f8a1 not found: ID does not exist" containerID="dc676a5de0132e297bd61c2b5704985d92b934eda8ed1a25a05619eb5ba0f8a1"
Oct 30 23:35:28.156184 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:28.155859    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc676a5de0132e297bd61c2b5704985d92b934eda8ed1a25a05619eb5ba0f8a1"} err="failed to get container status \"dc676a5de0132e297bd61c2b5704985d92b934eda8ed1a25a05619eb5ba0f8a1\": rpc error: code = NotFound desc = could not find container \"dc676a5de0132e297bd61c2b5704985d92b934eda8ed1a25a05619eb5ba0f8a1\": container with ID starting with dc676a5de0132e297bd61c2b5704985d92b934eda8ed1a25a05619eb5ba0f8a1 not found: ID does not exist"
Oct 30 23:35:28.156184 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:28.155882    5078 scope.go:122] "RemoveContainer" containerID="507a1dc6ac43513a08207acb61a5b7ac23838e3b2ffc993d75f01002fba6bce4"
Oct 30 23:35:28.156741 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:35:28.156589    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507a1dc6ac43513a08207acb61a5b7ac23838e3b2ffc993d75f01002fba6bce4\": container with ID starting with 507a1dc6ac43513a08207acb61a5b7ac23838e3b2ffc993d75f01002fba6bce4 not found: ID does not exist" containerID="507a1dc6ac43513a08207acb61a5b7ac23838e3b2ffc993d75f01002fba6bce4"
Oct 30 23:35:28.156741 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:28.156671    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507a1dc6ac43513a08207acb61a5b7ac23838e3b2ffc993d75f01002fba6bce4"} err="failed to get container status \"507a1dc6ac43513a08207acb61a5b7ac23838e3b2ffc993d75f01002fba6bce4\": rpc error: code = NotFound desc = could not find container \"507a1dc6ac43513a08207acb61a5b7ac23838e3b2ffc993d75f01002fba6bce4\": container with ID starting with 507a1dc6ac43513a08207acb61a5b7ac23838e3b2ffc993d75f01002fba6bce4 not found: ID does not exist"
Oct 30 23:35:28.373627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:28.372542    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"]
Oct 30 23:35:28.373627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:28.373148    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-20f33adb-gw-openshift-default-85b79f67d5-glftm"]
Oct 30 23:35:30.310437 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:30.310396    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" path="/var/lib/kubelet/pods/0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7/volumes"
Oct 30 23:35:32.563444 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:32.563408    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"]
Oct 30 23:35:32.564015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:32.563689    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc" podUID="b964ddd5-6455-412c-85a8-07cd6b15cf40" containerName="istio-proxy" containerID="cri-o://15609963fa4bc950ed64278e63d07beadd3fe5117ee162c78909feb234f4a781" gracePeriod=30
Oct 30 23:35:32.613499 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:32.613391    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"]
Oct 30 23:35:32.613830 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:32.613769    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j" podUID="3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" containerName="istio-proxy" containerID="cri-o://d039eeba52ec6ba8dce5bbdcb7e5dfc16bd0fd4f72c7d368fde0c8611de62583" gracePeriod=30
Oct 30 23:35:32.618590 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:32.618565    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"]
Oct 30 23:35:32.661771 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:32.661740    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"]
Oct 30 23:35:32.661967 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:32.661945    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm" podUID="24046e1c-b660-45a8-bb3c-d19c8b3ff167" containerName="istio-proxy" containerID="cri-o://bf9ca6c268ed31d755a6a78e241b189302e5ae903fb0950bd8e808dd3590e8ca" gracePeriod=30
Oct 30 23:35:33.136550 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:33.136515    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4" podUID="dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" containerName="istio-proxy" containerID="cri-o://41bab5387a9738ebcff4064abce5a1476516da88b53d79a2c6ec4cda43295493" gracePeriod=30
Oct 30 23:35:34.063552 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:34.063492    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:35:34.064486 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:34.064442    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm" podUID="24046e1c-b660-45a8-bb3c-d19c8b3ff167" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:35:37.339412 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:37.339375    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:35:37.339844 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:37.339445    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc" podUID="b964ddd5-6455-412c-85a8-07cd6b15cf40" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:35:37.345941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:37.345907    5078 patch_prober.go:29] interesting pod/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:35:37.346066 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:37.345953    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j" podUID="3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:35:38.107707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.107643    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:35:38.186233 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.186204    5078 generic.go:356] "Generic (PLEG): container finished" podID="3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" containerID="d039eeba52ec6ba8dce5bbdcb7e5dfc16bd0fd4f72c7d368fde0c8611de62583" exitCode=0
Oct 30 23:35:38.186380 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.186271    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j" event={"ID":"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97","Type":"ContainerDied","Data":"d039eeba52ec6ba8dce5bbdcb7e5dfc16bd0fd4f72c7d368fde0c8611de62583"}
Oct 30 23:35:38.188070 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.188045    5078 generic.go:356] "Generic (PLEG): container finished" podID="24046e1c-b660-45a8-bb3c-d19c8b3ff167" containerID="bf9ca6c268ed31d755a6a78e241b189302e5ae903fb0950bd8e808dd3590e8ca" exitCode=0
Oct 30 23:35:38.188209 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.188142    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm" event={"ID":"24046e1c-b660-45a8-bb3c-d19c8b3ff167","Type":"ContainerDied","Data":"bf9ca6c268ed31d755a6a78e241b189302e5ae903fb0950bd8e808dd3590e8ca"}
Oct 30 23:35:38.189713 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.189690    5078 generic.go:356] "Generic (PLEG): container finished" podID="b964ddd5-6455-412c-85a8-07cd6b15cf40" containerID="15609963fa4bc950ed64278e63d07beadd3fe5117ee162c78909feb234f4a781" exitCode=0
Oct 30 23:35:38.189825 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.189738    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"
Oct 30 23:35:38.189825 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.189744    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc" event={"ID":"b964ddd5-6455-412c-85a8-07cd6b15cf40","Type":"ContainerDied","Data":"15609963fa4bc950ed64278e63d07beadd3fe5117ee162c78909feb234f4a781"}
Oct 30 23:35:38.189825 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.189777    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc" event={"ID":"b964ddd5-6455-412c-85a8-07cd6b15cf40","Type":"ContainerDied","Data":"802fb72cca19d21574712615206444a18eb23bdb3f87bd8ec44bf2f89740a8f2"}
Oct 30 23:35:38.189825 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.189800    5078 scope.go:122] "RemoveContainer" containerID="15609963fa4bc950ed64278e63d07beadd3fe5117ee162c78909feb234f4a781"
Oct 30 23:35:38.242966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.241693    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-credential-socket\") pod \"b964ddd5-6455-412c-85a8-07cd6b15cf40\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") "
Oct 30 23:35:38.242966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.241792    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-envoy\") pod \"b964ddd5-6455-412c-85a8-07cd6b15cf40\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") "
Oct 30 23:35:38.242966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.241838    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-workload-socket\") pod \"b964ddd5-6455-412c-85a8-07cd6b15cf40\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") "
Oct 30 23:35:38.242966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.241927    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-token\" (UniqueName: \"kubernetes.io/projected/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-token\") pod \"b964ddd5-6455-412c-85a8-07cd6b15cf40\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") "
Oct 30 23:35:38.242966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.241994    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-workload-certs\") pod \"b964ddd5-6455-412c-85a8-07cd6b15cf40\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") "
Oct 30 23:35:38.242966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.242105    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-credential-socket" pod "b964ddd5-6455-412c-85a8-07cd6b15cf40" (UID: "b964ddd5-6455-412c-85a8-07cd6b15cf40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.242966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.242151    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b964ddd5-6455-412c-85a8-07cd6b15cf40-kube-api-access-c4pxv\" (UniqueName: \"kubernetes.io/projected/b964ddd5-6455-412c-85a8-07cd6b15cf40-kube-api-access-c4pxv\") pod \"b964ddd5-6455-412c-85a8-07cd6b15cf40\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") "
Oct 30 23:35:38.242966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.242323    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b964ddd5-6455-412c-85a8-07cd6b15cf40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b964ddd5-6455-412c-85a8-07cd6b15cf40-istiod-ca-cert\") pod \"b964ddd5-6455-412c-85a8-07cd6b15cf40\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") "
Oct 30 23:35:38.242966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.242359    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-podinfo\") pod \"b964ddd5-6455-412c-85a8-07cd6b15cf40\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") "
Oct 30 23:35:38.242966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.242391    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-data\") pod \"b964ddd5-6455-412c-85a8-07cd6b15cf40\" (UID: \"b964ddd5-6455-412c-85a8-07cd6b15cf40\") "
Oct 30 23:35:38.244933 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.243115    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b964ddd5-6455-412c-85a8-07cd6b15cf40-istiod-ca-cert" pod "b964ddd5-6455-412c-85a8-07cd6b15cf40" (UID: "b964ddd5-6455-412c-85a8-07cd6b15cf40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:35:38.244933 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.243438    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b964ddd5-6455-412c-85a8-07cd6b15cf40-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.244933 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.243445    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-data" pod "b964ddd5-6455-412c-85a8-07cd6b15cf40" (UID: "b964ddd5-6455-412c-85a8-07cd6b15cf40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.244933 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.243459    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.244933 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.243715    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-workload-certs" pod "b964ddd5-6455-412c-85a8-07cd6b15cf40" (UID: "b964ddd5-6455-412c-85a8-07cd6b15cf40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.244933 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.243734    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-workload-socket" pod "b964ddd5-6455-412c-85a8-07cd6b15cf40" (UID: "b964ddd5-6455-412c-85a8-07cd6b15cf40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.246039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.246005    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-envoy" pod "b964ddd5-6455-412c-85a8-07cd6b15cf40" (UID: "b964ddd5-6455-412c-85a8-07cd6b15cf40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.247440 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.247416    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-podinfo" pod "b964ddd5-6455-412c-85a8-07cd6b15cf40" (UID: "b964ddd5-6455-412c-85a8-07cd6b15cf40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:35:38.247911 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.247880    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b964ddd5-6455-412c-85a8-07cd6b15cf40-kube-api-access-c4pxv" pod "b964ddd5-6455-412c-85a8-07cd6b15cf40" (UID: "b964ddd5-6455-412c-85a8-07cd6b15cf40"). InnerVolumeSpecName "kube-api-access-c4pxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:35:38.248021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.247917    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-token" pod "b964ddd5-6455-412c-85a8-07cd6b15cf40" (UID: "b964ddd5-6455-412c-85a8-07cd6b15cf40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:35:38.254872 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.254856    5078 scope.go:122] "RemoveContainer" containerID="15609963fa4bc950ed64278e63d07beadd3fe5117ee162c78909feb234f4a781"
Oct 30 23:35:38.258274 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:35:38.258201    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15609963fa4bc950ed64278e63d07beadd3fe5117ee162c78909feb234f4a781\": container with ID starting with 15609963fa4bc950ed64278e63d07beadd3fe5117ee162c78909feb234f4a781 not found: ID does not exist" containerID="15609963fa4bc950ed64278e63d07beadd3fe5117ee162c78909feb234f4a781"
Oct 30 23:35:38.258274 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.258244    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15609963fa4bc950ed64278e63d07beadd3fe5117ee162c78909feb234f4a781"} err="failed to get container status \"15609963fa4bc950ed64278e63d07beadd3fe5117ee162c78909feb234f4a781\": rpc error: code = NotFound desc = could not find container \"15609963fa4bc950ed64278e63d07beadd3fe5117ee162c78909feb234f4a781\": container with ID starting with 15609963fa4bc950ed64278e63d07beadd3fe5117ee162c78909feb234f4a781 not found: ID does not exist"
Oct 30 23:35:38.262995 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.262423    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:35:38.265465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.265445    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:35:38.345170 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.345131    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.345170 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.345166    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.345552 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.345183    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.345552 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.345197    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.345552 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.345212    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4pxv\" (UniqueName: \"kubernetes.io/projected/b964ddd5-6455-412c-85a8-07cd6b15cf40-kube-api-access-c4pxv\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.345552 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.345228    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.345552 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.345240    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b964ddd5-6455-412c-85a8-07cd6b15cf40-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.446142 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.446057    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-workload-socket\") pod \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") "
Oct 30 23:35:38.446142 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.446123    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-podinfo\") pod \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") "
Oct 30 23:35:38.446383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.446175    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-podinfo\") pod \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") "
Oct 30 23:35:38.446383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.446210    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-workload-socket\") pod \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") "
Oct 30 23:35:38.446383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.446251    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-workload-certs\") pod \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") "
Oct 30 23:35:38.446383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.446304    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-token\" (UniqueName: \"kubernetes.io/projected/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-token\") pod \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") "
Oct 30 23:35:38.446383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.446373    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-envoy\") pod \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") "
Oct 30 23:35:38.446737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.446463    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-workload-socket" pod "3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" (UID: "3adadcee-68a6-4a1c-9c0c-d2fa9106ef97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.446737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.446487    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-kube-api-access-77jcx\" (UniqueName: \"kubernetes.io/projected/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-kube-api-access-77jcx\") pod \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") "
Oct 30 23:35:38.446737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.446656    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-workload-certs" pod "24046e1c-b660-45a8-bb3c-d19c8b3ff167" (UID: "24046e1c-b660-45a8-bb3c-d19c8b3ff167"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.446919 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.446585    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-credential-socket\") pod \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") "
Oct 30 23:35:38.446974 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.446925    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-envoy\") pod \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") "
Oct 30 23:35:38.446974 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.446883    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-workload-socket" pod "24046e1c-b660-45a8-bb3c-d19c8b3ff167" (UID: "24046e1c-b660-45a8-bb3c-d19c8b3ff167"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.447099 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.447080    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-token\" (UniqueName: \"kubernetes.io/projected/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-token\") pod \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") "
Oct 30 23:35:38.447231 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.447113    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istiod-ca-cert\") pod \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") "
Oct 30 23:35:38.447231 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.447119    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-credential-socket" pod "24046e1c-b660-45a8-bb3c-d19c8b3ff167" (UID: "24046e1c-b660-45a8-bb3c-d19c8b3ff167"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.447231 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.447133    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-credential-socket\") pod \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") "
Oct 30 23:35:38.447231 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.447163    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-workload-certs\") pod \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") "
Oct 30 23:35:38.447231 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.447180    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istiod-ca-cert\") pod \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") "
Oct 30 23:35:38.447506 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.447250    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-data\") pod \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") "
Oct 30 23:35:38.447506 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.447338    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-data\") pod \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\" (UID: \"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97\") "
Oct 30 23:35:38.447506 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.447394    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24046e1c-b660-45a8-bb3c-d19c8b3ff167-kube-api-access-572t9\" (UniqueName: \"kubernetes.io/projected/24046e1c-b660-45a8-bb3c-d19c8b3ff167-kube-api-access-572t9\") pod \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\" (UID: \"24046e1c-b660-45a8-bb3c-d19c8b3ff167\") "
Oct 30 23:35:38.448058 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.448022    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-workload-certs" pod "3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" (UID: "3adadcee-68a6-4a1c-9c0c-d2fa9106ef97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.448220 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.448194    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-data" pod "24046e1c-b660-45a8-bb3c-d19c8b3ff167" (UID: "24046e1c-b660-45a8-bb3c-d19c8b3ff167"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.448642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.448532    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istiod-ca-cert" pod "24046e1c-b660-45a8-bb3c-d19c8b3ff167" (UID: "24046e1c-b660-45a8-bb3c-d19c8b3ff167"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:35:38.448642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.448543    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.448642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.448576    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.448642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.448619    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.448642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.448632    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.448948 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.448648    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.448948 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.448661    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.448948 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.448778    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-data" pod "3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" (UID: "3adadcee-68a6-4a1c-9c0c-d2fa9106ef97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.448948 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.448844    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istiod-ca-cert" pod "3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" (UID: "3adadcee-68a6-4a1c-9c0c-d2fa9106ef97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:35:38.448948 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.448930    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-credential-socket" pod "3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" (UID: "3adadcee-68a6-4a1c-9c0c-d2fa9106ef97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.450300 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.450262    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-envoy" pod "24046e1c-b660-45a8-bb3c-d19c8b3ff167" (UID: "24046e1c-b660-45a8-bb3c-d19c8b3ff167"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.452545 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.452525    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-token" pod "3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" (UID: "3adadcee-68a6-4a1c-9c0c-d2fa9106ef97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:35:38.454045 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.454019    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-podinfo" pod "3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" (UID: "3adadcee-68a6-4a1c-9c0c-d2fa9106ef97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:35:38.454149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.454027    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-envoy" pod "3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" (UID: "3adadcee-68a6-4a1c-9c0c-d2fa9106ef97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.454149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.454073    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-kube-api-access-77jcx" pod "3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" (UID: "3adadcee-68a6-4a1c-9c0c-d2fa9106ef97"). InnerVolumeSpecName "kube-api-access-77jcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:35:38.454269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.454163    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24046e1c-b660-45a8-bb3c-d19c8b3ff167-kube-api-access-572t9" pod "24046e1c-b660-45a8-bb3c-d19c8b3ff167" (UID: "24046e1c-b660-45a8-bb3c-d19c8b3ff167"). InnerVolumeSpecName "kube-api-access-572t9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:35:38.455839 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.455813    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-podinfo" pod "24046e1c-b660-45a8-bb3c-d19c8b3ff167" (UID: "24046e1c-b660-45a8-bb3c-d19c8b3ff167"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:35:38.456158 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.456138    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-token" pod "24046e1c-b660-45a8-bb3c-d19c8b3ff167" (UID: "24046e1c-b660-45a8-bb3c-d19c8b3ff167"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:35:38.461817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.461794    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:35:38.549767 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.549731    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-workload-socket\") pod \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") "
Oct 30 23:35:38.549938 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.549830    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-credential-socket\") pod \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") "
Oct 30 23:35:38.549938 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.549877    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-data\") pod \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") "
Oct 30 23:35:38.550065 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.550010    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-workload-socket" pod "dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" (UID: "dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.550127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.550079    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-workload-certs\") pod \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") "
Oct 30 23:35:38.550187 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.550131    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-podinfo\") pod \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") "
Oct 30 23:35:38.550234 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.550203    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-credential-socket" pod "dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" (UID: "dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.550234 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.550210    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istiod-ca-cert\") pod \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") "
Oct 30 23:35:38.550326 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.550262    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-kube-api-access-4xlxl\" (UniqueName: \"kubernetes.io/projected/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-kube-api-access-4xlxl\") pod \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") "
Oct 30 23:35:38.550326 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.550300    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-token\" (UniqueName: \"kubernetes.io/projected/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-token\") pod \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") "
Oct 30 23:35:38.550326 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.550310    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-data" pod "dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" (UID: "dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.550472 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.550352    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-envoy\") pod \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\" (UID: \"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a\") "
Oct 30 23:35:38.550472 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.550403    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-workload-certs" pod "dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" (UID: "dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.550680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.550654    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istiod-ca-cert" pod "dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" (UID: "dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:35:38.551149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.551126    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.551303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.551289    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.551441 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.551425    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-572t9\" (UniqueName: \"kubernetes.io/projected/24046e1c-b660-45a8-bb3c-d19c8b3ff167-kube-api-access-572t9\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.551563 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.551549    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.551734 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.551719    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.551840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.551828    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.551948 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.551936    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.552039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.552028    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.552134 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.552122    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.552232 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.552213    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77jcx\" (UniqueName: \"kubernetes.io/projected/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-kube-api-access-77jcx\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.552334 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.552321    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.552425 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.552414    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.552515 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.552504    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.552729 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.552588    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24046e1c-b660-45a8-bb3c-d19c8b3ff167-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.552846 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.552831    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.552953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.552940    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.553075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.553062    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.553521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.553497    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-podinfo" pod "dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" (UID: "dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:35:38.553949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.553918    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-envoy" pod "dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" (UID: "dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:35:38.554096 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.554026    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-token" pod "dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" (UID: "dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:35:38.554555 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.554529    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-kube-api-access-4xlxl" pod "dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" (UID: "dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a"). InnerVolumeSpecName "kube-api-access-4xlxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:35:38.654678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.654638    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.654678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.654676    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xlxl\" (UniqueName: \"kubernetes.io/projected/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-kube-api-access-4xlxl\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.654897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.654693    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.654897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.654706    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:38.683722 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.683690    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"]
Oct 30 23:35:38.688878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:38.688853    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-shngc"]
Oct 30 23:35:39.200204 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.200175    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"
Oct 30 23:35:39.200404 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.200184    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j" event={"ID":"3adadcee-68a6-4a1c-9c0c-d2fa9106ef97","Type":"ContainerDied","Data":"3d437f6a9ebc8f9976de8595a682b4789c93482200f9674a06c4ba2a79b1f328"}
Oct 30 23:35:39.200404 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.200310    5078 scope.go:122] "RemoveContainer" containerID="d039eeba52ec6ba8dce5bbdcb7e5dfc16bd0fd4f72c7d368fde0c8611de62583"
Oct 30 23:35:39.202148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.202121    5078 generic.go:356] "Generic (PLEG): container finished" podID="dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" containerID="41bab5387a9738ebcff4064abce5a1476516da88b53d79a2c6ec4cda43295493" exitCode=0
Oct 30 23:35:39.202270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.202176    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"
Oct 30 23:35:39.202270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.202223    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4" event={"ID":"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a","Type":"ContainerDied","Data":"41bab5387a9738ebcff4064abce5a1476516da88b53d79a2c6ec4cda43295493"}
Oct 30 23:35:39.202270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.202269    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4" event={"ID":"dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a","Type":"ContainerDied","Data":"10650232309a3b097b202e43d8c79466a2565d160a91ed0b17bbaf1728af3f11"}
Oct 30 23:35:39.204790 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.204762    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm" event={"ID":"24046e1c-b660-45a8-bb3c-d19c8b3ff167","Type":"ContainerDied","Data":"b71a0d47f18defc21e5498b264586e73de9fbf1ea42ff925da2560bea8db6427"}
Oct 30 23:35:39.204916 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.204774    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"
Oct 30 23:35:39.214020 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.214005    5078 scope.go:122] "RemoveContainer" containerID="41bab5387a9738ebcff4064abce5a1476516da88b53d79a2c6ec4cda43295493"
Oct 30 23:35:39.223983 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.223962    5078 scope.go:122] "RemoveContainer" containerID="41bab5387a9738ebcff4064abce5a1476516da88b53d79a2c6ec4cda43295493"
Oct 30 23:35:39.224245 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:35:39.224227    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41bab5387a9738ebcff4064abce5a1476516da88b53d79a2c6ec4cda43295493\": container with ID starting with 41bab5387a9738ebcff4064abce5a1476516da88b53d79a2c6ec4cda43295493 not found: ID does not exist" containerID="41bab5387a9738ebcff4064abce5a1476516da88b53d79a2c6ec4cda43295493"
Oct 30 23:35:39.224295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.224253    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41bab5387a9738ebcff4064abce5a1476516da88b53d79a2c6ec4cda43295493"} err="failed to get container status \"41bab5387a9738ebcff4064abce5a1476516da88b53d79a2c6ec4cda43295493\": rpc error: code = NotFound desc = could not find container \"41bab5387a9738ebcff4064abce5a1476516da88b53d79a2c6ec4cda43295493\": container with ID starting with 41bab5387a9738ebcff4064abce5a1476516da88b53d79a2c6ec4cda43295493 not found: ID does not exist"
Oct 30 23:35:39.224295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.224270    5078 scope.go:122] "RemoveContainer" containerID="bf9ca6c268ed31d755a6a78e241b189302e5ae903fb0950bd8e808dd3590e8ca"
Oct 30 23:35:39.365971 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.365937    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"]
Oct 30 23:35:39.372477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.372455    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-6tl6j"]
Oct 30 23:35:39.482944 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.482868    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"]
Oct 30 23:35:39.486209 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.486190    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-kccm4"]
Oct 30 23:35:39.615619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.612731    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"]
Oct 30 23:35:39.621375 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:39.621267    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-989a4f-openshift-default-697f765b6c-wgrqm"]
Oct 30 23:35:40.301474 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:40.301434    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24046e1c-b660-45a8-bb3c-d19c8b3ff167" path="/var/lib/kubelet/pods/24046e1c-b660-45a8-bb3c-d19c8b3ff167/volumes"
Oct 30 23:35:40.302088 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:40.302068    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" path="/var/lib/kubelet/pods/3adadcee-68a6-4a1c-9c0c-d2fa9106ef97/volumes"
Oct 30 23:35:40.302526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:40.302510    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b964ddd5-6455-412c-85a8-07cd6b15cf40" path="/var/lib/kubelet/pods/b964ddd5-6455-412c-85a8-07cd6b15cf40/volumes"
Oct 30 23:35:40.303418 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:40.303401    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" path="/var/lib/kubelet/pods/dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a/volumes"
Oct 30 23:35:47.592586 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:35:47.592552    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:35:50.089070 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:50.089036    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx"]
Oct 30 23:35:50.138192 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:50.138161    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fbbb3083-z2hpx"]
Oct 30 23:35:50.299667 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:50.299633    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f5e60c9-45fa-479f-8166-142e02a4ec18" path="/var/lib/kubelet/pods/9f5e60c9-45fa-479f-8166-142e02a4ec18/volumes"
Oct 30 23:35:53.941287 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.941248    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf"]
Oct 30 23:35:53.941934 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.941917    5078 cpu_manager.go:414] "containerMap: removing container" podUID="24046e1c-b660-45a8-bb3c-d19c8b3ff167" containerName="istio-proxy"
Oct 30 23:35:53.941978 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.941935    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24046e1c-b660-45a8-bb3c-d19c8b3ff167" containerName="istio-proxy"
Oct 30 23:35:53.941978 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.941948    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" containerName="istio-proxy"
Oct 30 23:35:53.941978 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.941953    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" containerName="istio-proxy"
Oct 30 23:35:53.941978 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.941974    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" containerName="istio-proxy"
Oct 30 23:35:53.942104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.941979    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" containerName="istio-proxy"
Oct 30 23:35:53.942104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.941999    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b964ddd5-6455-412c-85a8-07cd6b15cf40" containerName="istio-proxy"
Oct 30 23:35:53.942104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.942004    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b964ddd5-6455-412c-85a8-07cd6b15cf40" containerName="istio-proxy"
Oct 30 23:35:53.942104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.942016    5078 cpu_manager.go:414] "containerMap: removing container" podUID="dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" containerName="istio-proxy"
Oct 30 23:35:53.942104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.942020    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" containerName="istio-proxy"
Oct 30 23:35:53.942250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.942128    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3adadcee-68a6-4a1c-9c0c-d2fa9106ef97" containerName="istio-proxy"
Oct 30 23:35:53.942250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.942141    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" containerName="istio-proxy"
Oct 30 23:35:53.942250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.942148    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b964ddd5-6455-412c-85a8-07cd6b15cf40" containerName="istio-proxy"
Oct 30 23:35:53.942250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.942156    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd03d3b6-2d7b-4f65-97c8-a4ea4384d80a" containerName="istio-proxy"
Oct 30 23:35:53.942250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.942162    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="24046e1c-b660-45a8-bb3c-d19c8b3ff167" containerName="istio-proxy"
Oct 30 23:35:53.945829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.945812    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf"
Oct 30 23:35:53.961545 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.961518    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffd9j\" (UniqueName: \"kubernetes.io/projected/88c35deb-c0ef-4790-ac5b-725649bba7b5-kube-api-access-ffd9j\") pod \"maas-api-revoke-sub-bc285e6e-qndpf\" (UID: \"88c35deb-c0ef-4790-ac5b-725649bba7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf"
Oct 30 23:35:53.961694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.961574    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c35deb-c0ef-4790-ac5b-725649bba7b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc285e6e-qndpf\" (UID: \"88c35deb-c0ef-4790-ac5b-725649bba7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf"
Oct 30 23:35:53.961772 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:53.961759    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c35deb-c0ef-4790-ac5b-725649bba7b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc285e6e-qndpf\" (UID: \"88c35deb-c0ef-4790-ac5b-725649bba7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf"
Oct 30 23:35:54.002884 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:54.002849    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:35:54.004574 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:54.004557    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:35:54.052927 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:54.052891    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:35:54.053982 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:54.053966    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:35:54.058015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:54.057988    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf"]
Oct 30 23:35:54.063118 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:54.063094    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffd9j\" (UniqueName: \"kubernetes.io/projected/88c35deb-c0ef-4790-ac5b-725649bba7b5-kube-api-access-ffd9j\") pod \"maas-api-revoke-sub-bc285e6e-qndpf\" (UID: \"88c35deb-c0ef-4790-ac5b-725649bba7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf"
Oct 30 23:35:54.063248 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:54.063149    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c35deb-c0ef-4790-ac5b-725649bba7b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc285e6e-qndpf\" (UID: \"88c35deb-c0ef-4790-ac5b-725649bba7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf"
Oct 30 23:35:54.063316 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:54.063296    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c35deb-c0ef-4790-ac5b-725649bba7b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc285e6e-qndpf\" (UID: \"88c35deb-c0ef-4790-ac5b-725649bba7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf"
Oct 30 23:35:54.064090 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:54.064062    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c35deb-c0ef-4790-ac5b-725649bba7b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc285e6e-qndpf\" (UID: \"88c35deb-c0ef-4790-ac5b-725649bba7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf"
Oct 30 23:35:54.065845 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:54.065820    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c35deb-c0ef-4790-ac5b-725649bba7b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc285e6e-qndpf\" (UID: \"88c35deb-c0ef-4790-ac5b-725649bba7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf"
Oct 30 23:35:54.125357 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:54.125322    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffd9j\" (UniqueName: \"kubernetes.io/projected/88c35deb-c0ef-4790-ac5b-725649bba7b5-kube-api-access-ffd9j\") pod \"maas-api-revoke-sub-bc285e6e-qndpf\" (UID: \"88c35deb-c0ef-4790-ac5b-725649bba7b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf"
Oct 30 23:35:54.315063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:54.315027    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:35:54.321165 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:54.321145    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf"
Oct 30 23:35:54.705342 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:35:54.705309    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c35deb_c0ef_4790_ac5b_725649bba7b5.slice/crio-3ccd961f20254972cb7a7009f39e020b0806e83f33fe58c8f9e844f66c6f0e20 WatchSource:0}: Error finding container 3ccd961f20254972cb7a7009f39e020b0806e83f33fe58c8f9e844f66c6f0e20: Status 404 returned error can't find the container with id 3ccd961f20254972cb7a7009f39e020b0806e83f33fe58c8f9e844f66c6f0e20
Oct 30 23:35:54.706048 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:54.706021    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf"]
Oct 30 23:35:55.633917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:55.633883    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf" event={"ID":"88c35deb-c0ef-4790-ac5b-725649bba7b5","Type":"ContainerStarted","Data":"56196ad0921e1f9edbd0c17112f0e8d9ac08e167e3670541f81a4ade7d4c403b"}
Oct 30 23:35:55.636533 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:55.636298    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf" event={"ID":"88c35deb-c0ef-4790-ac5b-725649bba7b5","Type":"ContainerStarted","Data":"3ccd961f20254972cb7a7009f39e020b0806e83f33fe58c8f9e844f66c6f0e20"}
Oct 30 23:35:55.752246 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:55.752190    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf" podStartSLOduration=2.752176305 podStartE2EDuration="2.752176305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:35:55.750253849 +0000 UTC m=+1741.638160215" watchObservedRunningTime="2026-10-30 23:35:55.752176305 +0000 UTC m=+1741.640082657"
Oct 30 23:35:56.643551 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:56.643512    5078 generic.go:356] "Generic (PLEG): container finished" podID="88c35deb-c0ef-4790-ac5b-725649bba7b5" containerID="56196ad0921e1f9edbd0c17112f0e8d9ac08e167e3670541f81a4ade7d4c403b" exitCode=0
Oct 30 23:35:56.643998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:56.643637    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf" event={"ID":"88c35deb-c0ef-4790-ac5b-725649bba7b5","Type":"ContainerDied","Data":"56196ad0921e1f9edbd0c17112f0e8d9ac08e167e3670541f81a4ade7d4c403b"}
Oct 30 23:35:57.889731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:57.889709    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf"
Oct 30 23:35:57.975976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:57.975904    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88c35deb-c0ef-4790-ac5b-725649bba7b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c35deb-c0ef-4790-ac5b-725649bba7b5-maas-api-openshift-ca\") pod \"88c35deb-c0ef-4790-ac5b-725649bba7b5\" (UID: \"88c35deb-c0ef-4790-ac5b-725649bba7b5\") "
Oct 30 23:35:57.975976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:57.975946    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88c35deb-c0ef-4790-ac5b-725649bba7b5-kube-api-access-ffd9j\" (UniqueName: \"kubernetes.io/projected/88c35deb-c0ef-4790-ac5b-725649bba7b5-kube-api-access-ffd9j\") pod \"88c35deb-c0ef-4790-ac5b-725649bba7b5\" (UID: \"88c35deb-c0ef-4790-ac5b-725649bba7b5\") "
Oct 30 23:35:57.976173 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:57.976003    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88c35deb-c0ef-4790-ac5b-725649bba7b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c35deb-c0ef-4790-ac5b-725649bba7b5-maas-api-xks-ca\") pod \"88c35deb-c0ef-4790-ac5b-725649bba7b5\" (UID: \"88c35deb-c0ef-4790-ac5b-725649bba7b5\") "
Oct 30 23:35:57.976297 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:57.976272    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c35deb-c0ef-4790-ac5b-725649bba7b5-maas-api-openshift-ca" pod "88c35deb-c0ef-4790-ac5b-725649bba7b5" (UID: "88c35deb-c0ef-4790-ac5b-725649bba7b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:35:57.976777 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:57.976754    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c35deb-c0ef-4790-ac5b-725649bba7b5-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:57.978159 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:57.978137    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c35deb-c0ef-4790-ac5b-725649bba7b5-maas-api-xks-ca" pod "88c35deb-c0ef-4790-ac5b-725649bba7b5" (UID: "88c35deb-c0ef-4790-ac5b-725649bba7b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:35:57.978289 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:57.978264    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c35deb-c0ef-4790-ac5b-725649bba7b5-kube-api-access-ffd9j" pod "88c35deb-c0ef-4790-ac5b-725649bba7b5" (UID: "88c35deb-c0ef-4790-ac5b-725649bba7b5"). InnerVolumeSpecName "kube-api-access-ffd9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:35:58.077804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:58.077776    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffd9j\" (UniqueName: \"kubernetes.io/projected/88c35deb-c0ef-4790-ac5b-725649bba7b5-kube-api-access-ffd9j\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:58.077804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:58.077804    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c35deb-c0ef-4790-ac5b-725649bba7b5-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:35:58.722000 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:58.721925    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf"
Oct 30 23:35:58.722000 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:58.721963    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf" event={"ID":"88c35deb-c0ef-4790-ac5b-725649bba7b5","Type":"ContainerDied","Data":"3ccd961f20254972cb7a7009f39e020b0806e83f33fe58c8f9e844f66c6f0e20"}
Oct 30 23:35:58.722268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:35:58.722012    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ccd961f20254972cb7a7009f39e020b0806e83f33fe58c8f9e844f66c6f0e20"
Oct 30 23:36:01.742833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:01.742803    5078 scope.go:122] "RemoveContainer" containerID="a59207809e954a4a06d2a1832d86b66864d7b1929d464a1a8c88bd8d6590d468"
Oct 30 23:36:01.756834 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:01.756812    5078 scope.go:122] "RemoveContainer" containerID="ef81c2a1084c193a898e06dfd054906b097221921ff53710b58881d9527f5579"
Oct 30 23:36:01.766061 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:01.766043    5078 scope.go:122] "RemoveContainer" containerID="8f71cc26fd13dffad980af1693d0dbcd346263b40ed6e98c5fe6a466a489c007"
Oct 30 23:36:01.775116 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:01.775088    5078 scope.go:122] "RemoveContainer" containerID="6d2c4db6a5c435e90da35bf26c2a58fa44a99ac56992d2d4946a6f79d0cdedc2"
Oct 30 23:36:01.784422 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:01.784406    5078 scope.go:122] "RemoveContainer" containerID="166731348a496c6251b00645ad8de451e8386c0bbe43af67c02a23e9c02cac9e"
Oct 30 23:36:01.793403 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:01.793386    5078 scope.go:122] "RemoveContainer" containerID="ceb6b951397bb18b3771a576a7ce94c7ef5174458ecc095744ec2de072400afd"
Oct 30 23:36:01.801873 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:01.801854    5078 scope.go:122] "RemoveContainer" containerID="761ccb188dc52e99a71f16b9768a034642cb215dad7d282fa767d625ed5ea6e6"
Oct 30 23:36:01.810332 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:01.810317    5078 scope.go:122] "RemoveContainer" containerID="0f4551819a075b72f8d0bb5ee020de67f82c33e774f939facd75229f2718cffe"
Oct 30 23:36:01.819510 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:01.819491    5078 scope.go:122] "RemoveContainer" containerID="0674c02fba5a87566bcc6aff9cef3d2b02117feb92c5893b4e52495cea4a537d"
Oct 30 23:36:01.827864 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:01.827846    5078 scope.go:122] "RemoveContainer" containerID="2ea4be0290714ae52635e0972fae244fb82054ce74d1265345dcbcdf89784a91"
Oct 30 23:36:01.836390 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:01.836373    5078 scope.go:122] "RemoveContainer" containerID="a1fa1868ba803443e529227ad1cca3c89819546af8b179f861de2e94acfb8415"
Oct 30 23:36:01.845047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:01.845028    5078 scope.go:122] "RemoveContainer" containerID="e4b98e17f546d49aa4db0eec3a2166038b8bc3042a8250b3d608e5ce4a736a4c"
Oct 30 23:36:17.597280 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:36:17.597237    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:36:44.598208 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.598167    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"]
Oct 30 23:36:44.598809 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.598797    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" containerName="istio-proxy"
Oct 30 23:36:44.598809 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.598810    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" containerName="istio-proxy"
Oct 30 23:36:44.598941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.598835    5078 cpu_manager.go:414] "containerMap: removing container" podUID="88c35deb-c0ef-4790-ac5b-725649bba7b5" containerName="revoke-keys"
Oct 30 23:36:44.598941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.598840    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88c35deb-c0ef-4790-ac5b-725649bba7b5" containerName="revoke-keys"
Oct 30 23:36:44.599011 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.598973    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b655ccf-f1c3-43e8-82b7-259f9a9cf0e7" containerName="istio-proxy"
Oct 30 23:36:44.599011 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.598983    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="88c35deb-c0ef-4790-ac5b-725649bba7b5" containerName="revoke-keys"
Oct 30 23:36:44.602545 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.602526    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.638919 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.638881    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-envoy\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.638919 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.638921    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-token\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.639140 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.638971    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-credential-socket\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.639140 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.639018    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-workload-socket\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.639140 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.639044    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpdzv\" (UniqueName: \"kubernetes.io/projected/c8b4ee1d-90dd-416a-9955-cc54ad064723-kube-api-access-lpdzv\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.639300 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.639153    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-data\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.639300 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.639185    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-podinfo\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.639300 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.639211    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-workload-certs\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.639300 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.639293    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b4ee1d-90dd-416a-9955-cc54ad064723-istiod-ca-cert\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.654715 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.654681    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-3be34b-openshift-default-dockercfg-w4lgv\""
Oct 30 23:36:44.710443 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.710411    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"]
Oct 30 23:36:44.740799 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.740767    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-data\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.741004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.740808    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-podinfo\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.741004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.740832    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-workload-certs\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.741004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.740927    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b4ee1d-90dd-416a-9955-cc54ad064723-istiod-ca-cert\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.741004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.741004    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-envoy\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.741261 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.741030    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-token\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.741261 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.741130    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-credential-socket\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.741261 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.741192    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-workload-socket\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.741261 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.741222    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpdzv\" (UniqueName: \"kubernetes.io/projected/c8b4ee1d-90dd-416a-9955-cc54ad064723-kube-api-access-lpdzv\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.741485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.741255    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-data\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.741485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.741431    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-workload-certs\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.741632 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.741512    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-credential-socket\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.741786 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.741761    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-workload-socket\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.742224 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.742161    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b4ee1d-90dd-416a-9955-cc54ad064723-istiod-ca-cert\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.744101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.744071    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-envoy\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.744189 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.744085    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-podinfo\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.802443 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.802410    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-token\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.802663 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.802623    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpdzv\" (UniqueName: \"kubernetes.io/projected/c8b4ee1d-90dd-416a-9955-cc54ad064723-kube-api-access-lpdzv\") pod \"e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") " pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:44.927675 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:44.922066    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:45.493523 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:45.493497    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"]
Oct 30 23:36:45.493918 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:36:45.493890    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b4ee1d_90dd_416a_9955_cc54ad064723.slice/crio-d4348bfcc56bbab9da49811ed7495a76be6775175b069934462e48e4cc5dc6a3 WatchSource:0}: Error finding container d4348bfcc56bbab9da49811ed7495a76be6775175b069934462e48e4cc5dc6a3: Status 404 returned error can't find the container with id d4348bfcc56bbab9da49811ed7495a76be6775175b069934462e48e4cc5dc6a3
Oct 30 23:36:45.496608 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:45.496565    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:36:45.496684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:45.496662    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:36:45.496730 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:45.496696    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:36:46.138826 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:46.138789    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c" event={"ID":"c8b4ee1d-90dd-416a-9955-cc54ad064723","Type":"ContainerStarted","Data":"12b24b5e11c1cd12348c0a391bd1e210ffc895b878b5be170495658ea95ddb13"}
Oct 30 23:36:46.139336 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:46.138853    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c" event={"ID":"c8b4ee1d-90dd-416a-9955-cc54ad064723","Type":"ContainerStarted","Data":"d4348bfcc56bbab9da49811ed7495a76be6775175b069934462e48e4cc5dc6a3"}
Oct 30 23:36:46.314720 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:46.314663    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c" podStartSLOduration=2.314648047 podStartE2EDuration="2.314648047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:36:46.308919141 +0000 UTC m=+1792.196825495" watchObservedRunningTime="2026-10-30 23:36:46.314648047 +0000 UTC m=+1792.202554399"
Oct 30 23:36:46.922450 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:46.922420    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:46.928195 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:46.928172    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:47.145687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:47.145652    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:47.146457 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:47.146439    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:36:47.715017 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:36:47.714983    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:36:55.283570 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:55.283524    5078 generic.go:356] "Generic (PLEG): container finished" podID="776ad186-9f08-44e9-99e6-f9bc82acc93c" containerID="bf9135ff1b93ab4e7a8f6f0ac5ba38b4e8ae5bc4c48c83566f3ceb0b3810d72a" exitCode=0
Oct 30 23:36:55.284121 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:55.283649    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd" event={"ID":"776ad186-9f08-44e9-99e6-f9bc82acc93c","Type":"ContainerDied","Data":"bf9135ff1b93ab4e7a8f6f0ac5ba38b4e8ae5bc4c48c83566f3ceb0b3810d72a"}
Oct 30 23:36:55.284243 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:55.284223    5078 scope.go:122] "RemoveContainer" containerID="bf9135ff1b93ab4e7a8f6f0ac5ba38b4e8ae5bc4c48c83566f3ceb0b3810d72a"
Oct 30 23:36:55.286168 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:55.286142    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:36:55.286297 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:55.286211    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:36:55.286297 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:55.286243    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:36:56.423622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:56.413183    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd" event={"ID":"776ad186-9f08-44e9-99e6-f9bc82acc93c","Type":"ContainerStarted","Data":"5971a2020d2b4ea109b0fab3b87954c6a320e94003f601ce2ae06af3cecbea35"}
Oct 30 23:36:56.454135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:56.454100    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:36:56.455312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:56.455291    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 23:36:56.455366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:56.455337    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd" podUID="776ad186-9f08-44e9-99e6-f9bc82acc93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 23:36:57.459621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:57.456328    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 23:36:57.459621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:57.456392    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd" podUID="776ad186-9f08-44e9-99e6-f9bc82acc93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 23:36:58.454183 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:58.454151    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 23:36:58.454366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:58.454212    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd" podUID="776ad186-9f08-44e9-99e6-f9bc82acc93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 23:36:58.484749 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:58.484660    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:36:59.732641 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:59.721251    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 23:36:59.732641 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:36:59.721326    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd" podUID="776ad186-9f08-44e9-99e6-f9bc82acc93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 23:37:00.475276 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:00.474013    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 23:37:00.475276 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:00.474082    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd" podUID="776ad186-9f08-44e9-99e6-f9bc82acc93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 23:37:01.455161 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.455116    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 23:37:01.455680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.455179    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd" podUID="776ad186-9f08-44e9-99e6-f9bc82acc93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 23:37:01.682622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.679996    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"]
Oct 30 23:37:01.701946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.701914    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:01.755699 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.721589    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-podinfo\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:01.755699 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.721679    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-workload-socket\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:01.755699 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.721767    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-envoy\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:01.755699 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.721806    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-workload-certs\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:01.755699 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.722002    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-token\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:01.755699 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.722104    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-data\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:01.755699 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.722234    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7f4356-b694-4ebb-a65e-8f7782591aea-istiod-ca-cert\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:01.755699 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.722343    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58cdz\" (UniqueName: \"kubernetes.io/projected/eb7f4356-b694-4ebb-a65e-8f7782591aea-kube-api-access-58cdz\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:01.755699 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.722390    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-credential-socket\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:01.766848 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.765569    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"]
Oct 30 23:37:01.803642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.781838    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-bcc619-openshift-default-dockercfg-w9z88\""
Oct 30 23:37:01.946971 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.946932    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-credential-socket\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:01.948751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.948713    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-podinfo\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:01.966042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.947894    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-credential-socket\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:01.966436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.966406    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-workload-socket\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:01.967265 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.967053    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-workload-socket\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:01.969423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:01.969399    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-envoy\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:02.048707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.030846    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-workload-certs\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:02.048707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.031031    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-token\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:02.048707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.031059    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-data\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:02.048707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.031147    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7f4356-b694-4ebb-a65e-8f7782591aea-istiod-ca-cert\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:02.048707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.031272    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58cdz\" (UniqueName: \"kubernetes.io/projected/eb7f4356-b694-4ebb-a65e-8f7782591aea-kube-api-access-58cdz\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:02.048707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.032012    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-workload-certs\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:02.048707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.032501    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-data\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:02.048707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.033716    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7f4356-b694-4ebb-a65e-8f7782591aea-istiod-ca-cert\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:02.048707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.035102    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-podinfo\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:02.156924 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.141490    5078 scope.go:122] "RemoveContainer" containerID="4e28b566dbf46ce2c526ea15e9621d214b344da4e8680dbb8bb907f4ee9e9f24"
Oct 30 23:37:02.174628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.174315    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58cdz\" (UniqueName: \"kubernetes.io/projected/eb7f4356-b694-4ebb-a65e-8f7782591aea-kube-api-access-58cdz\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:02.254622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.251545    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-envoy\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:02.273233 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.273136    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-token\") pod \"e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") " pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:02.425634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.419890    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:02.455275 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.455135    5078 patch_prober.go:29] interesting pod/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body=
Oct 30 23:37:02.456310 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.455691    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd" podUID="776ad186-9f08-44e9-99e6-f9bc82acc93c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused"
Oct 30 23:37:02.533456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.532070    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/ready\": context deadline exceeded"
Oct 30 23:37:02.554203 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.550223    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:37:02.591400 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:02.590681    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn" podUID="7160e014-9b36-4fc3-afc6-82aa41241b3d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.21:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:37:03.012999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:03.012973    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"]
Oct 30 23:37:03.014995 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:37:03.014969    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7f4356_b694_4ebb_a65e_8f7782591aea.slice/crio-66486bbcd57086f4b6a7c5fce7c2b52c480103a35e0425b0163f53103ee5a4e1 WatchSource:0}: Error finding container 66486bbcd57086f4b6a7c5fce7c2b52c480103a35e0425b0163f53103ee5a4e1: Status 404 returned error can't find the container with id 66486bbcd57086f4b6a7c5fce7c2b52c480103a35e0425b0163f53103ee5a4e1
Oct 30 23:37:03.018170 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:03.018139    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:37:03.018259 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:03.018208    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:37:03.018259 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:03.018238    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:37:03.037701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:03.037678    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4" event={"ID":"eb7f4356-b694-4ebb-a65e-8f7782591aea","Type":"ContainerStarted","Data":"66486bbcd57086f4b6a7c5fce7c2b52c480103a35e0425b0163f53103ee5a4e1"}
Oct 30 23:37:03.554503 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:03.554440    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:37:03.556419 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:03.556397    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:37:04.048367 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:04.048317    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4" event={"ID":"eb7f4356-b694-4ebb-a65e-8f7782591aea","Type":"ContainerStarted","Data":"f430980ddda5807584ae8cf577187e1ee40beb081b49c4c82c969495242684b9"}
Oct 30 23:37:04.168357 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:04.168304    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4" podStartSLOduration=3.168287916 podStartE2EDuration="3.168287916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:04.166482439 +0000 UTC m=+1810.054388805" watchObservedRunningTime="2026-10-30 23:37:04.168287916 +0000 UTC m=+1810.056194270"
Oct 30 23:37:04.421170 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:04.421086    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:04.426832 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:04.426810    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:05.056748 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:05.056721    5078 generic.go:356] "Generic (PLEG): container finished" podID="0b13c626-bbb0-4874-88b2-cecb77a61423" containerID="e813ebaa8a7b9cdc591c908ab710b4afc85a904f813c38e3c7d98a99eb3079b9" exitCode=0
Oct 30 23:37:05.057471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:05.056796    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz" event={"ID":"0b13c626-bbb0-4874-88b2-cecb77a61423","Type":"ContainerDied","Data":"e813ebaa8a7b9cdc591c908ab710b4afc85a904f813c38e3c7d98a99eb3079b9"}
Oct 30 23:37:05.057471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:05.057157    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:05.057471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:05.057236    5078 scope.go:122] "RemoveContainer" containerID="e813ebaa8a7b9cdc591c908ab710b4afc85a904f813c38e3c7d98a99eb3079b9"
Oct 30 23:37:05.058239 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:05.058033    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:37:05.058903 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:05.058876    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:37:05.058999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:05.058942    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:37:05.058999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:05.058976    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:37:05.549172 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:05.549144    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:37:06.090694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:06.090654    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz" event={"ID":"0b13c626-bbb0-4874-88b2-cecb77a61423","Type":"ContainerStarted","Data":"6f6623d2d72a32cfb982b097626d6693f122f62d9ffe0cdf3ca0d458329a12d5"}
Oct 30 23:37:06.762968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:06.762937    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:37:06.766752 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:06.766722    5078 patch_prober.go:29] interesting pod/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 23:37:06.767005 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:06.766974    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz" podUID="0b13c626-bbb0-4874-88b2-cecb77a61423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 23:37:07.778384 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:07.777951    5078 patch_prober.go:29] interesting pod/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 23:37:07.778384 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:07.778010    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz" podUID="0b13c626-bbb0-4874-88b2-cecb77a61423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 23:37:08.941621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:08.938047    5078 patch_prober.go:29] interesting pod/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 23:37:08.941621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:08.938118    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz" podUID="0b13c626-bbb0-4874-88b2-cecb77a61423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 23:37:09.781625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:09.780707    5078 patch_prober.go:29] interesting pod/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 23:37:09.781625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:09.780788    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz" podUID="0b13c626-bbb0-4874-88b2-cecb77a61423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 23:37:10.759341 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:10.759305    5078 patch_prober.go:29] interesting pod/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused" start-of-body=
Oct 30 23:37:10.759968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:10.759936    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz" podUID="0b13c626-bbb0-4874-88b2-cecb77a61423" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.156:15021/healthz/ready\": dial tcp 10.133.0.156:15021: connect: connection refused"
Oct 30 23:37:11.844901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:11.844866    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:37:11.845869 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:11.845845    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:37:11.846687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:11.846663    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:37:14.807503 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:14.807473    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7384-31fa34"]
Oct 30 23:37:14.811793 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:14.811771    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-31fa34"
Oct 30 23:37:14.861373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:14.861331    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-625gk\" (UniqueName: \"kubernetes.io/projected/f67b2423-0e70-4058-a595-7f2976e4c609-kube-api-access-625gk\") pod \"test-curl-7384-31fa34\" (UID: \"f67b2423-0e70-4058-a595-7f2976e4c609\") " pod="opendatahub/test-curl-7384-31fa34"
Oct 30 23:37:14.864155 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:14.864129    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 23:37:14.866802 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:14.866778    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-31fa34"]
Oct 30 23:37:14.912175 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:14.912146    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 23:37:14.931005 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:14.930979    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qwr4w\""
Oct 30 23:37:14.963039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:14.963011    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-625gk\" (UniqueName: \"kubernetes.io/projected/f67b2423-0e70-4058-a595-7f2976e4c609-kube-api-access-625gk\") pod \"test-curl-7384-31fa34\" (UID: \"f67b2423-0e70-4058-a595-7f2976e4c609\") " pod="opendatahub/test-curl-7384-31fa34"
Oct 30 23:37:15.020139 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:15.020109    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-625gk\" (UniqueName: \"kubernetes.io/projected/f67b2423-0e70-4058-a595-7f2976e4c609-kube-api-access-625gk\") pod \"test-curl-7384-31fa34\" (UID: \"f67b2423-0e70-4058-a595-7f2976e4c609\") " pod="opendatahub/test-curl-7384-31fa34"
Oct 30 23:37:15.125266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:15.125197    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-31fa34"
Oct 30 23:37:15.505765 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:15.505733    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-31fa34"]
Oct 30 23:37:15.506881 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:37:15.506851    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf67b2423_0e70_4058_a595_7f2976e4c609.slice/crio-f16fd11f141d8ae5631912d5bdc1e9a21fbcc82e743350cf12aa3e5c2103f2ec WatchSource:0}: Error finding container f16fd11f141d8ae5631912d5bdc1e9a21fbcc82e743350cf12aa3e5c2103f2ec: Status 404 returned error can't find the container with id f16fd11f141d8ae5631912d5bdc1e9a21fbcc82e743350cf12aa3e5c2103f2ec
Oct 30 23:37:15.509382 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:15.509359    5078 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:37:15.528351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:15.528329    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-31fa34" event={"ID":"f67b2423-0e70-4058-a595-7f2976e4c609","Type":"ContainerStarted","Data":"f16fd11f141d8ae5631912d5bdc1e9a21fbcc82e743350cf12aa3e5c2103f2ec"}
Oct 30 23:37:17.599401 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:37:17.599362    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:37:18.552205 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:18.552171    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-31fa34" event={"ID":"f67b2423-0e70-4058-a595-7f2976e4c609","Type":"ContainerStarted","Data":"117e4646d4deeb18f773504b8b867137f31336d416c473fb76e960a540d0083a"}
Oct 30 23:37:18.668469 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:18.668416    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7384-31fa34" podStartSLOduration=2.110858198 podStartE2EDuration="4.668402383s" totalImagesPullingTime="2.557544185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:18.664471384 +0000 UTC m=+1824.552377737" watchObservedRunningTime="2026-10-30 23:37:18.668402383 +0000 UTC m=+1824.556308736"
Oct 30 23:37:20.114027 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:20.113947    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7384-31fa34"]
Oct 30 23:37:20.130676 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:20.130642    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7384-31fa34"]
Oct 30 23:37:20.266198 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:20.266167    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7384-b2670d"]
Oct 30 23:37:20.266864 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:20.266848    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f67b2423-0e70-4058-a595-7f2976e4c609" containerName="test-curl-7384-31fa34"
Oct 30 23:37:20.266923 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:20.266866    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f67b2423-0e70-4058-a595-7f2976e4c609" containerName="test-curl-7384-31fa34"
Oct 30 23:37:20.266989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:20.266979    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f67b2423-0e70-4058-a595-7f2976e4c609" containerName="test-curl-7384-31fa34"
Oct 30 23:37:20.270548 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:20.270533    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-b2670d"
Oct 30 23:37:20.324516 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:20.324491    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-b2670d"]
Oct 30 23:37:20.354927 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:20.354886    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7jsr\" (UniqueName: \"kubernetes.io/projected/a68fdc10-ada8-4c4b-9efd-60cf43b8f5af-kube-api-access-j7jsr\") pod \"test-curl-7384-b2670d\" (UID: \"a68fdc10-ada8-4c4b-9efd-60cf43b8f5af\") " pod="opendatahub/test-curl-7384-b2670d"
Oct 30 23:37:20.456850 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:20.456764    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7jsr\" (UniqueName: \"kubernetes.io/projected/a68fdc10-ada8-4c4b-9efd-60cf43b8f5af-kube-api-access-j7jsr\") pod \"test-curl-7384-b2670d\" (UID: \"a68fdc10-ada8-4c4b-9efd-60cf43b8f5af\") " pod="opendatahub/test-curl-7384-b2670d"
Oct 30 23:37:20.513541 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:20.513508    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7jsr\" (UniqueName: \"kubernetes.io/projected/a68fdc10-ada8-4c4b-9efd-60cf43b8f5af-kube-api-access-j7jsr\") pod \"test-curl-7384-b2670d\" (UID: \"a68fdc10-ada8-4c4b-9efd-60cf43b8f5af\") " pod="opendatahub/test-curl-7384-b2670d"
Oct 30 23:37:20.571723 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:20.571669    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7384-31fa34" podUID="f67b2423-0e70-4058-a595-7f2976e4c609" containerName="test-curl-7384-31fa34" containerID="cri-o://117e4646d4deeb18f773504b8b867137f31336d416c473fb76e960a540d0083a" gracePeriod=2
Oct 30 23:37:20.583312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:20.583272    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-b2670d"
Oct 30 23:37:20.960246 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:20.960218    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-b2670d"]
Oct 30 23:37:20.960246 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:37:20.960234    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda68fdc10_ada8_4c4b_9efd_60cf43b8f5af.slice/crio-ee43e9f0710e723cc17aa3525ba8ed01b7d23280c62693d18c89ec93ff2438f5 WatchSource:0}: Error finding container ee43e9f0710e723cc17aa3525ba8ed01b7d23280c62693d18c89ec93ff2438f5: Status 404 returned error can't find the container with id ee43e9f0710e723cc17aa3525ba8ed01b7d23280c62693d18c89ec93ff2438f5
Oct 30 23:37:21.587882 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:21.587847    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-b2670d" event={"ID":"a68fdc10-ada8-4c4b-9efd-60cf43b8f5af","Type":"ContainerStarted","Data":"65be169850e092d18396826655f2f639e31b490335437ec5a50d44d3bec1d5c0"}
Oct 30 23:37:21.588325 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:21.587894    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-b2670d" event={"ID":"a68fdc10-ada8-4c4b-9efd-60cf43b8f5af","Type":"ContainerStarted","Data":"ee43e9f0710e723cc17aa3525ba8ed01b7d23280c62693d18c89ec93ff2438f5"}
Oct 30 23:37:21.706892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:21.706843    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7384-b2670d" podStartSLOduration=1.706829664 podStartE2EDuration="1.706829664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:21.705379333 +0000 UTC m=+1827.593285686" watchObservedRunningTime="2026-10-30 23:37:21.706829664 +0000 UTC m=+1827.594736016"
Oct 30 23:37:22.819688 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:22.819662    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-31fa34"
Oct 30 23:37:22.994720 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:22.994649    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f67b2423-0e70-4058-a595-7f2976e4c609-kube-api-access-625gk\" (UniqueName: \"kubernetes.io/projected/f67b2423-0e70-4058-a595-7f2976e4c609-kube-api-access-625gk\") pod \"f67b2423-0e70-4058-a595-7f2976e4c609\" (UID: \"f67b2423-0e70-4058-a595-7f2976e4c609\") "
Oct 30 23:37:22.996684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:22.996662    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f67b2423-0e70-4058-a595-7f2976e4c609-kube-api-access-625gk" pod "f67b2423-0e70-4058-a595-7f2976e4c609" (UID: "f67b2423-0e70-4058-a595-7f2976e4c609"). InnerVolumeSpecName "kube-api-access-625gk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:37:23.096839 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.096813    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-625gk\" (UniqueName: \"kubernetes.io/projected/f67b2423-0e70-4058-a595-7f2976e4c609-kube-api-access-625gk\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:37:23.176171 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.176144    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7384-b2670d"]
Oct 30 23:37:23.180810 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.180774    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7384-b2670d"]
Oct 30 23:37:23.335543 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.335497    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7384-5538ad"]
Oct 30 23:37:23.336200 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.336179    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a68fdc10-ada8-4c4b-9efd-60cf43b8f5af" containerName="test-curl-7384-b2670d"
Oct 30 23:37:23.336325 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.336201    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a68fdc10-ada8-4c4b-9efd-60cf43b8f5af" containerName="test-curl-7384-b2670d"
Oct 30 23:37:23.336426 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.336412    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a68fdc10-ada8-4c4b-9efd-60cf43b8f5af" containerName="test-curl-7384-b2670d"
Oct 30 23:37:23.339878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.339859    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-5538ad"
Oct 30 23:37:23.393807 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.393782    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-5538ad"]
Oct 30 23:37:23.503070 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.503043    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qwrj\" (UniqueName: \"kubernetes.io/projected/9a43406f-f310-460f-8f85-4bae4f7fd2bc-kube-api-access-6qwrj\") pod \"test-curl-7384-5538ad\" (UID: \"9a43406f-f310-460f-8f85-4bae4f7fd2bc\") " pod="opendatahub/test-curl-7384-5538ad"
Oct 30 23:37:23.603888 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.603806    5078 generic.go:356] "Generic (PLEG): container finished" podID="f67b2423-0e70-4058-a595-7f2976e4c609" containerID="117e4646d4deeb18f773504b8b867137f31336d416c473fb76e960a540d0083a" exitCode=137
Oct 30 23:37:23.603888 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.603842    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-31fa34"
Oct 30 23:37:23.604107 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.603901    5078 scope.go:122] "RemoveContainer" containerID="117e4646d4deeb18f773504b8b867137f31336d416c473fb76e960a540d0083a"
Oct 30 23:37:23.604107 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.604089    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7384-b2670d" podUID="a68fdc10-ada8-4c4b-9efd-60cf43b8f5af" containerName="test-curl-7384-b2670d" containerID="cri-o://65be169850e092d18396826655f2f639e31b490335437ec5a50d44d3bec1d5c0" gracePeriod=2
Oct 30 23:37:23.604222 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.604178    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qwrj\" (UniqueName: \"kubernetes.io/projected/9a43406f-f310-460f-8f85-4bae4f7fd2bc-kube-api-access-6qwrj\") pod \"test-curl-7384-5538ad\" (UID: \"9a43406f-f310-460f-8f85-4bae4f7fd2bc\") " pod="opendatahub/test-curl-7384-5538ad"
Oct 30 23:37:23.615803 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.615785    5078 scope.go:122] "RemoveContainer" containerID="117e4646d4deeb18f773504b8b867137f31336d416c473fb76e960a540d0083a"
Oct 30 23:37:23.616063 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:37:23.616046    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"117e4646d4deeb18f773504b8b867137f31336d416c473fb76e960a540d0083a\": container with ID starting with 117e4646d4deeb18f773504b8b867137f31336d416c473fb76e960a540d0083a not found: ID does not exist" containerID="117e4646d4deeb18f773504b8b867137f31336d416c473fb76e960a540d0083a"
Oct 30 23:37:23.616119 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.616071    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"117e4646d4deeb18f773504b8b867137f31336d416c473fb76e960a540d0083a"} err="failed to get container status \"117e4646d4deeb18f773504b8b867137f31336d416c473fb76e960a540d0083a\": rpc error: code = NotFound desc = could not find container \"117e4646d4deeb18f773504b8b867137f31336d416c473fb76e960a540d0083a\": container with ID starting with 117e4646d4deeb18f773504b8b867137f31336d416c473fb76e960a540d0083a not found: ID does not exist"
Oct 30 23:37:23.655256 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:37:23.655227    5078 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-b2670d\" is forbidden: User \"system:node:ip-10-0-130-205.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-205.us-west-1.compute.internal' and this object" podUID="a68fdc10-ada8-4c4b-9efd-60cf43b8f5af" pod="opendatahub/test-curl-7384-b2670d"
Oct 30 23:37:23.661442 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.661421    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qwrj\" (UniqueName: \"kubernetes.io/projected/9a43406f-f310-460f-8f85-4bae4f7fd2bc-kube-api-access-6qwrj\") pod \"test-curl-7384-5538ad\" (UID: \"9a43406f-f310-460f-8f85-4bae4f7fd2bc\") " pod="opendatahub/test-curl-7384-5538ad"
Oct 30 23:37:23.706134 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:37:23.706108    5078 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-31fa34\" is forbidden: User \"system:node:ip-10-0-130-205.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-205.us-west-1.compute.internal' and this object" podUID="f67b2423-0e70-4058-a595-7f2976e4c609" pod="opendatahub/test-curl-7384-31fa34"
Oct 30 23:37:23.757685 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:37:23.757657    5078 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-b2670d\" is forbidden: User \"system:node:ip-10-0-130-205.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-205.us-west-1.compute.internal' and this object" podUID="a68fdc10-ada8-4c4b-9efd-60cf43b8f5af" pod="opendatahub/test-curl-7384-b2670d"
Oct 30 23:37:23.951716 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:23.951625    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-5538ad"
Oct 30 23:37:24.047988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:24.047956    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc"]
Oct 30 23:37:24.095150 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:24.095116    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8de42e8c-ftxzc"]
Oct 30 23:37:24.302889 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:24.302852    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02e3dd03-03e7-42cb-a6b6-3680f039b67b" path="/var/lib/kubelet/pods/02e3dd03-03e7-42cb-a6b6-3680f039b67b/volumes"
Oct 30 23:37:24.303576 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:24.303553    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f67b2423-0e70-4058-a595-7f2976e4c609" path="/var/lib/kubelet/pods/f67b2423-0e70-4058-a595-7f2976e4c609/volumes"
Oct 30 23:37:24.327362 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:24.327340    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-5538ad"]
Oct 30 23:37:24.328160 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:37:24.328132    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a43406f_f310_460f_8f85_4bae4f7fd2bc.slice/crio-0b07b2046b1257ca190949459b02545e3fd7558859e80b51e6fbcdd18aaa48a1 WatchSource:0}: Error finding container 0b07b2046b1257ca190949459b02545e3fd7558859e80b51e6fbcdd18aaa48a1: Status 404 returned error can't find the container with id 0b07b2046b1257ca190949459b02545e3fd7558859e80b51e6fbcdd18aaa48a1
Oct 30 23:37:24.346311 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:37:24.346287    5078 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-31fa34\" is forbidden: User \"system:node:ip-10-0-130-205.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-205.us-west-1.compute.internal' and this object" podUID="f67b2423-0e70-4058-a595-7f2976e4c609" pod="opendatahub/test-curl-7384-31fa34"
Oct 30 23:37:24.396938 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:37:24.396905    5078 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-b2670d\" is forbidden: User \"system:node:ip-10-0-130-205.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-205.us-west-1.compute.internal' and this object" podUID="a68fdc10-ada8-4c4b-9efd-60cf43b8f5af" pod="opendatahub/test-curl-7384-b2670d"
Oct 30 23:37:24.613115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:24.613028    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-5538ad" event={"ID":"9a43406f-f310-460f-8f85-4bae4f7fd2bc","Type":"ContainerStarted","Data":"f1b538252edd3e42488426f405384de44c6e25ce50b1efc3d28619882186cc1e"}
Oct 30 23:37:24.613115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:24.613068    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-5538ad" event={"ID":"9a43406f-f310-460f-8f85-4bae4f7fd2bc","Type":"ContainerStarted","Data":"0b07b2046b1257ca190949459b02545e3fd7558859e80b51e6fbcdd18aaa48a1"}
Oct 30 23:37:24.726929 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:24.726875    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7384-5538ad" podStartSLOduration=1.726860501 podStartE2EDuration="1.726860501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:24.723738951 +0000 UTC m=+1830.611645307" watchObservedRunningTime="2026-10-30 23:37:24.726860501 +0000 UTC m=+1830.614766853"
Oct 30 23:37:25.868660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:25.868574    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-b2670d"
Oct 30 23:37:26.039178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.039143    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a68fdc10-ada8-4c4b-9efd-60cf43b8f5af-kube-api-access-j7jsr\" (UniqueName: \"kubernetes.io/projected/a68fdc10-ada8-4c4b-9efd-60cf43b8f5af-kube-api-access-j7jsr\") pod \"a68fdc10-ada8-4c4b-9efd-60cf43b8f5af\" (UID: \"a68fdc10-ada8-4c4b-9efd-60cf43b8f5af\") "
Oct 30 23:37:26.041480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.041450    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a68fdc10-ada8-4c4b-9efd-60cf43b8f5af-kube-api-access-j7jsr" pod "a68fdc10-ada8-4c4b-9efd-60cf43b8f5af" (UID: "a68fdc10-ada8-4c4b-9efd-60cf43b8f5af"). InnerVolumeSpecName "kube-api-access-j7jsr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:37:26.117656 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.117625    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7384-5538ad"]
Oct 30 23:37:26.122748 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.122678    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7384-5538ad"]
Oct 30 23:37:26.142002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.141963    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7jsr\" (UniqueName: \"kubernetes.io/projected/a68fdc10-ada8-4c4b-9efd-60cf43b8f5af-kube-api-access-j7jsr\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:37:26.258258 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.258226    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7384-7e4f84"]
Oct 30 23:37:26.258897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.258881    5078 cpu_manager.go:414] "containerMap: removing container" podUID="9a43406f-f310-460f-8f85-4bae4f7fd2bc" containerName="test-curl-7384-5538ad"
Oct 30 23:37:26.258953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.258899    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a43406f-f310-460f-8f85-4bae4f7fd2bc" containerName="test-curl-7384-5538ad"
Oct 30 23:37:26.259072 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.259062    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a43406f-f310-460f-8f85-4bae4f7fd2bc" containerName="test-curl-7384-5538ad"
Oct 30 23:37:26.262421 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.262405    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-7e4f84"
Oct 30 23:37:26.298974 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.298951    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a68fdc10-ada8-4c4b-9efd-60cf43b8f5af" path="/var/lib/kubelet/pods/a68fdc10-ada8-4c4b-9efd-60cf43b8f5af/volumes"
Oct 30 23:37:26.316005 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.315985    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-7e4f84"]
Oct 30 23:37:26.447243 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.447158    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbd2p\" (UniqueName: \"kubernetes.io/projected/0e7837ab-5482-4a86-bbb4-09310753d6fb-kube-api-access-lbd2p\") pod \"test-curl-7384-7e4f84\" (UID: \"0e7837ab-5482-4a86-bbb4-09310753d6fb\") " pod="opendatahub/test-curl-7384-7e4f84"
Oct 30 23:37:26.548421 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.548389    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbd2p\" (UniqueName: \"kubernetes.io/projected/0e7837ab-5482-4a86-bbb4-09310753d6fb-kube-api-access-lbd2p\") pod \"test-curl-7384-7e4f84\" (UID: \"0e7837ab-5482-4a86-bbb4-09310753d6fb\") " pod="opendatahub/test-curl-7384-7e4f84"
Oct 30 23:37:26.605411 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.605373    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbd2p\" (UniqueName: \"kubernetes.io/projected/0e7837ab-5482-4a86-bbb4-09310753d6fb-kube-api-access-lbd2p\") pod \"test-curl-7384-7e4f84\" (UID: \"0e7837ab-5482-4a86-bbb4-09310753d6fb\") " pod="opendatahub/test-curl-7384-7e4f84"
Oct 30 23:37:26.629634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.629572    5078 generic.go:356] "Generic (PLEG): container finished" podID="a68fdc10-ada8-4c4b-9efd-60cf43b8f5af" containerID="65be169850e092d18396826655f2f639e31b490335437ec5a50d44d3bec1d5c0" exitCode=137
Oct 30 23:37:26.629634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.629632    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-b2670d"
Oct 30 23:37:26.629833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.629651    5078 scope.go:122] "RemoveContainer" containerID="65be169850e092d18396826655f2f639e31b490335437ec5a50d44d3bec1d5c0"
Oct 30 23:37:26.629833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.629766    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7384-5538ad" podUID="9a43406f-f310-460f-8f85-4bae4f7fd2bc" containerName="test-curl-7384-5538ad" containerID="cri-o://f1b538252edd3e42488426f405384de44c6e25ce50b1efc3d28619882186cc1e" gracePeriod=2
Oct 30 23:37:26.639859 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.639844    5078 scope.go:122] "RemoveContainer" containerID="65be169850e092d18396826655f2f639e31b490335437ec5a50d44d3bec1d5c0"
Oct 30 23:37:26.640094 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:37:26.640070    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65be169850e092d18396826655f2f639e31b490335437ec5a50d44d3bec1d5c0\": container with ID starting with 65be169850e092d18396826655f2f639e31b490335437ec5a50d44d3bec1d5c0 not found: ID does not exist" containerID="65be169850e092d18396826655f2f639e31b490335437ec5a50d44d3bec1d5c0"
Oct 30 23:37:26.640149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.640100    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65be169850e092d18396826655f2f639e31b490335437ec5a50d44d3bec1d5c0"} err="failed to get container status \"65be169850e092d18396826655f2f639e31b490335437ec5a50d44d3bec1d5c0\": rpc error: code = NotFound desc = could not find container \"65be169850e092d18396826655f2f639e31b490335437ec5a50d44d3bec1d5c0\": container with ID starting with 65be169850e092d18396826655f2f639e31b490335437ec5a50d44d3bec1d5c0 not found: ID does not exist"
Oct 30 23:37:26.875046 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:26.875012    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-7e4f84"
Oct 30 23:37:27.256719 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:27.256691    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7384-7e4f84"]
Oct 30 23:37:27.257349 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:37:27.257316    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e7837ab_5482_4a86_bbb4_09310753d6fb.slice/crio-f9e31184fce09ddc97c34389e9a9552cb5d345cd46d440090fe8ceefdc5c1582 WatchSource:0}: Error finding container f9e31184fce09ddc97c34389e9a9552cb5d345cd46d440090fe8ceefdc5c1582: Status 404 returned error can't find the container with id f9e31184fce09ddc97c34389e9a9552cb5d345cd46d440090fe8ceefdc5c1582
Oct 30 23:37:27.638282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:27.638245    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-7e4f84" event={"ID":"0e7837ab-5482-4a86-bbb4-09310753d6fb","Type":"ContainerStarted","Data":"3664d022427d148f772789e378b555b12c8a4e9e27a885fbb2a140421485f27c"}
Oct 30 23:37:27.638487 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:27.638295    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7384-7e4f84" event={"ID":"0e7837ab-5482-4a86-bbb4-09310753d6fb","Type":"ContainerStarted","Data":"f9e31184fce09ddc97c34389e9a9552cb5d345cd46d440090fe8ceefdc5c1582"}
Oct 30 23:37:27.753544 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:27.753490    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7384-7e4f84" podStartSLOduration=1.75347505 podStartE2EDuration="1.75347505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:27.749799635 +0000 UTC m=+1833.637705990" watchObservedRunningTime="2026-10-30 23:37:27.75347505 +0000 UTC m=+1833.641381403"
Oct 30 23:37:28.883713 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:28.883692    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-5538ad"
Oct 30 23:37:28.990270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:28.990198    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a43406f-f310-460f-8f85-4bae4f7fd2bc-kube-api-access-6qwrj\" (UniqueName: \"kubernetes.io/projected/9a43406f-f310-460f-8f85-4bae4f7fd2bc-kube-api-access-6qwrj\") pod \"9a43406f-f310-460f-8f85-4bae4f7fd2bc\" (UID: \"9a43406f-f310-460f-8f85-4bae4f7fd2bc\") "
Oct 30 23:37:28.992361 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:28.992335    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a43406f-f310-460f-8f85-4bae4f7fd2bc-kube-api-access-6qwrj" pod "9a43406f-f310-460f-8f85-4bae4f7fd2bc" (UID: "9a43406f-f310-460f-8f85-4bae4f7fd2bc"). InnerVolumeSpecName "kube-api-access-6qwrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:37:29.092188 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:29.092163    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qwrj\" (UniqueName: \"kubernetes.io/projected/9a43406f-f310-460f-8f85-4bae4f7fd2bc-kube-api-access-6qwrj\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:37:29.205214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:29.205171    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7384-7e4f84"]
Oct 30 23:37:29.251149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:29.251077    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7384-7e4f84"]
Oct 30 23:37:29.679605 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:29.679557    5078 generic.go:356] "Generic (PLEG): container finished" podID="9a43406f-f310-460f-8f85-4bae4f7fd2bc" containerID="f1b538252edd3e42488426f405384de44c6e25ce50b1efc3d28619882186cc1e" exitCode=137
Oct 30 23:37:29.679605 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:29.679605    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-5538ad"
Oct 30 23:37:29.679826 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:29.679642    5078 scope.go:122] "RemoveContainer" containerID="f1b538252edd3e42488426f405384de44c6e25ce50b1efc3d28619882186cc1e"
Oct 30 23:37:29.679986 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:29.679939    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7384-7e4f84" podUID="0e7837ab-5482-4a86-bbb4-09310753d6fb" containerName="test-curl-7384-7e4f84" containerID="cri-o://3664d022427d148f772789e378b555b12c8a4e9e27a885fbb2a140421485f27c" gracePeriod=2
Oct 30 23:37:29.690560 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:29.690539    5078 scope.go:122] "RemoveContainer" containerID="f1b538252edd3e42488426f405384de44c6e25ce50b1efc3d28619882186cc1e"
Oct 30 23:37:29.690846 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:37:29.690828    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1b538252edd3e42488426f405384de44c6e25ce50b1efc3d28619882186cc1e\": container with ID starting with f1b538252edd3e42488426f405384de44c6e25ce50b1efc3d28619882186cc1e not found: ID does not exist" containerID="f1b538252edd3e42488426f405384de44c6e25ce50b1efc3d28619882186cc1e"
Oct 30 23:37:29.690911 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:29.690857    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1b538252edd3e42488426f405384de44c6e25ce50b1efc3d28619882186cc1e"} err="failed to get container status \"f1b538252edd3e42488426f405384de44c6e25ce50b1efc3d28619882186cc1e\": rpc error: code = NotFound desc = could not find container \"f1b538252edd3e42488426f405384de44c6e25ce50b1efc3d28619882186cc1e\": container with ID starting with f1b538252edd3e42488426f405384de44c6e25ce50b1efc3d28619882186cc1e not found: ID does not exist"
Oct 30 23:37:29.730769 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:37:29.730740    5078 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-7e4f84\" is forbidden: User \"system:node:ip-10-0-130-205.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-205.us-west-1.compute.internal' and this object" podUID="0e7837ab-5482-4a86-bbb4-09310753d6fb" pod="opendatahub/test-curl-7384-7e4f84"
Oct 30 23:37:29.781414 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:37:29.781384    5078 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-5538ad\" is forbidden: User \"system:node:ip-10-0-130-205.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-205.us-west-1.compute.internal' and this object" podUID="9a43406f-f310-460f-8f85-4bae4f7fd2bc" pod="opendatahub/test-curl-7384-5538ad"
Oct 30 23:37:29.832059 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:37:29.832029    5078 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-7e4f84\" is forbidden: User \"system:node:ip-10-0-130-205.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-205.us-west-1.compute.internal' and this object" podUID="0e7837ab-5482-4a86-bbb4-09310753d6fb" pod="opendatahub/test-curl-7384-7e4f84"
Oct 30 23:37:30.321813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.319730    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"]
Oct 30 23:37:30.321813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.320755    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0e7837ab-5482-4a86-bbb4-09310753d6fb" containerName="test-curl-7384-7e4f84"
Oct 30 23:37:30.321813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.320775    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e7837ab-5482-4a86-bbb4-09310753d6fb" containerName="test-curl-7384-7e4f84"
Oct 30 23:37:30.321813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.321046    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e7837ab-5482-4a86-bbb4-09310753d6fb" containerName="test-curl-7384-7e4f84"
Oct 30 23:37:30.819626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.818766    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.856058 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.856023    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a43406f-f310-460f-8f85-4bae4f7fd2bc" path="/var/lib/kubelet/pods/9a43406f-f310-460f-8f85-4bae4f7fd2bc/volumes"
Oct 30 23:37:30.856465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.856443    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"]
Oct 30 23:37:30.877688 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.877668    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b526ac-openshift-default-dockercfg-74hq9\""
Oct 30 23:37:30.878425 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:37:30.878401    5078 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-7e4f84\" is forbidden: User \"system:node:ip-10-0-130-205.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-205.us-west-1.compute.internal' and this object" podUID="0e7837ab-5482-4a86-bbb4-09310753d6fb" pod="opendatahub/test-curl-7384-7e4f84"
Oct 30 23:37:30.884329 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.884304    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bca2f196-2daf-4071-b366-596e7ff0f23d-istiod-ca-cert\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.884456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.884436    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-envoy\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.884499 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.884476    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-workload-socket\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.884539 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.884529    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-workload-certs\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.884773 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.884746    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-credential-socket\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.884820 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.884790    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-data\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.884910 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.884896    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsnkv\" (UniqueName: \"kubernetes.io/projected/bca2f196-2daf-4071-b366-596e7ff0f23d-kube-api-access-xsnkv\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.884945 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.884938    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-token\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.885067 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.885049    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-podinfo\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.986586 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.986553    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-workload-certs\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.986805 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.986659    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-credential-socket\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.986805 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.986705    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-data\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.986805 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.986775    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsnkv\" (UniqueName: \"kubernetes.io/projected/bca2f196-2daf-4071-b366-596e7ff0f23d-kube-api-access-xsnkv\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.986986 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.986844    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-token\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.986986 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.986954    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-podinfo\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.987093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.986988    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bca2f196-2daf-4071-b366-596e7ff0f23d-istiod-ca-cert\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.987093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.987036    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-workload-certs\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.987229 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.987087    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-credential-socket\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.987229 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.987153    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-data\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.987229 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.987205    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-envoy\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.987386 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.987265    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-workload-socket\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.987685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.987660    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-workload-socket\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.987842 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.987692    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bca2f196-2daf-4071-b366-596e7ff0f23d-istiod-ca-cert\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.989701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.989678    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-envoy\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:30.989940 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:30.989925    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-podinfo\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:31.053087 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:31.053058    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsnkv\" (UniqueName: \"kubernetes.io/projected/bca2f196-2daf-4071-b366-596e7ff0f23d-kube-api-access-xsnkv\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:31.054178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:31.054152    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-token\") pod \"e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") " pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:31.060175 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:37:31.060145    5078 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7384-5538ad\" is forbidden: User \"system:node:ip-10-0-130-205.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-205.us-west-1.compute.internal' and this object" podUID="9a43406f-f310-460f-8f85-4bae4f7fd2bc" pod="opendatahub/test-curl-7384-5538ad"
Oct 30 23:37:31.158119 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:31.158022    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:31.601030 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:37:31.600997    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca2f196_2daf_4071_b366_596e7ff0f23d.slice/crio-9717568152c15da37403a3d2319c2694819727d04334c632d28828d46fe1fd20 WatchSource:0}: Error finding container 9717568152c15da37403a3d2319c2694819727d04334c632d28828d46fe1fd20: Status 404 returned error can't find the container with id 9717568152c15da37403a3d2319c2694819727d04334c632d28828d46fe1fd20
Oct 30 23:37:31.601427 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:31.601372    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"]
Oct 30 23:37:31.604317 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:31.604282    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:37:31.604416 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:31.604362    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:37:31.604416 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:31.604402    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:37:32.001671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:32.001091    5078 generic.go:356] "Generic (PLEG): container finished" podID="0e7837ab-5482-4a86-bbb4-09310753d6fb" containerID="3664d022427d148f772789e378b555b12c8a4e9e27a885fbb2a140421485f27c" exitCode=137
Oct 30 23:37:32.032654 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:32.024710    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg" event={"ID":"bca2f196-2daf-4071-b366-596e7ff0f23d","Type":"ContainerStarted","Data":"866067e7d180f1fefe39467b336b5e9a398c807897dd7dd24fd4c0fe6d92182a"}
Oct 30 23:37:32.032654 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:32.024781    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg" event={"ID":"bca2f196-2daf-4071-b366-596e7ff0f23d","Type":"ContainerStarted","Data":"9717568152c15da37403a3d2319c2694819727d04334c632d28828d46fe1fd20"}
Oct 30 23:37:32.165720 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:32.159956    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:32.165720 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:32.163532    5078 patch_prober.go:29] interesting pod/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 23:37:32.165720 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:32.163625    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg" podUID="bca2f196-2daf-4071-b366-596e7ff0f23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 23:37:32.180962 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:32.177203    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg" podStartSLOduration=2.177182164 podStartE2EDuration="2.177182164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:32.158539052 +0000 UTC m=+1838.046445406" watchObservedRunningTime="2026-10-30 23:37:32.177182164 +0000 UTC m=+1838.065088522"
Oct 30 23:37:32.237389 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:32.237352    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-7e4f84"
Oct 30 23:37:32.310774 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:32.310743    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e7837ab-5482-4a86-bbb4-09310753d6fb-kube-api-access-lbd2p\" (UniqueName: \"kubernetes.io/projected/0e7837ab-5482-4a86-bbb4-09310753d6fb-kube-api-access-lbd2p\") pod \"0e7837ab-5482-4a86-bbb4-09310753d6fb\" (UID: \"0e7837ab-5482-4a86-bbb4-09310753d6fb\") "
Oct 30 23:37:32.313157 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:32.313119    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7837ab-5482-4a86-bbb4-09310753d6fb-kube-api-access-lbd2p" pod "0e7837ab-5482-4a86-bbb4-09310753d6fb" (UID: "0e7837ab-5482-4a86-bbb4-09310753d6fb"). InnerVolumeSpecName "kube-api-access-lbd2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:37:32.430623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:32.423193    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbd2p\" (UniqueName: \"kubernetes.io/projected/0e7837ab-5482-4a86-bbb4-09310753d6fb-kube-api-access-lbd2p\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:37:33.033475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:33.033446    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7384-7e4f84"
Oct 30 23:37:33.033993 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:33.033508    5078 scope.go:122] "RemoveContainer" containerID="3664d022427d148f772789e378b555b12c8a4e9e27a885fbb2a140421485f27c"
Oct 30 23:37:33.163930 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:33.163905    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:34.042145 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:34.042118    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:34.043236 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:34.043218    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:37:34.300791 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:34.300719    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e7837ab-5482-4a86-bbb4-09310753d6fb" path="/var/lib/kubelet/pods/0e7837ab-5482-4a86-bbb4-09310753d6fb/volumes"
Oct 30 23:37:47.593629 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:37:47.593569    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:37:52.432378 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.432344    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"]
Oct 30 23:37:52.515157 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.515130    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.584613 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.575715    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-7a42d8-openshift-default-dockercfg-ww9hq\""
Oct 30 23:37:52.584613 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.581297    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"]
Oct 30 23:37:52.618634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.618578    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-workload-socket\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.624023 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.624000    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-workload-certs\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.625673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.625248    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-podinfo\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.626390 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.626366    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-envoy\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.626612 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.626573    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-token\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.626846 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.626827    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-data\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.627073 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.627053    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68c5r\" (UniqueName: \"kubernetes.io/projected/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-kube-api-access-68c5r\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.627278 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.627259    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-credential-socket\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.627445 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.627426    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istiod-ca-cert\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.733103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.731777    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-envoy\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.733103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.731830    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-token\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.733103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.731889    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-data\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.733103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.731942    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68c5r\" (UniqueName: \"kubernetes.io/projected/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-kube-api-access-68c5r\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.733103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.732005    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-credential-socket\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.733103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.732030    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istiod-ca-cert\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.733103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.732238    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-workload-socket\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.733103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.732284    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-workload-certs\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.733103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.732337    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-podinfo\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.744969 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.744936    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istiod-ca-cert\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.746761 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.745823    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-credential-socket\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.746954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.746177    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-data\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.747069 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.746432    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-workload-socket\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.747408 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.746667    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-workload-certs\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.788042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.779287    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-podinfo\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.801168 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.793296    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68c5r\" (UniqueName: \"kubernetes.io/projected/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-kube-api-access-68c5r\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.812538 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.812000    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-envoy\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.822047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.822019    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-token\") pod \"e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") " pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:52.926936 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:52.926894    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:53.437673 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:37:53.437642    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cf62568_28ad_43c6_a0ab_f1680ef1a72c.slice/crio-e40bbeb65759e87ddabee11c1a5d9217c96cba624209364fa83daaf167d6acfa WatchSource:0}: Error finding container e40bbeb65759e87ddabee11c1a5d9217c96cba624209364fa83daaf167d6acfa: Status 404 returned error can't find the container with id e40bbeb65759e87ddabee11c1a5d9217c96cba624209364fa83daaf167d6acfa
Oct 30 23:37:53.438104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:53.437836    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"]
Oct 30 23:37:53.440558 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:53.440528    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:37:53.440659 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:53.440644    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:37:53.440709 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:53.440678    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:37:54.228314 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:54.228280    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv" event={"ID":"4cf62568-28ad-43c6-a0ab-f1680ef1a72c","Type":"ContainerStarted","Data":"dadf087ab6b027c3f6a6f2cc1e9e2daf54de156826205181c8f938b27e1afc16"}
Oct 30 23:37:54.228514 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:54.228339    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv" event={"ID":"4cf62568-28ad-43c6-a0ab-f1680ef1a72c","Type":"ContainerStarted","Data":"e40bbeb65759e87ddabee11c1a5d9217c96cba624209364fa83daaf167d6acfa"}
Oct 30 23:37:54.927207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:54.927181    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:54.933162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:54.933138    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:55.053364 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:55.053318    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv" podStartSLOduration=3.053304653 podStartE2EDuration="3.053304653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:37:54.443337636 +0000 UTC m=+1860.331244000" watchObservedRunningTime="2026-10-30 23:37:55.053304653 +0000 UTC m=+1860.941211078"
Oct 30 23:37:55.236232 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:55.236149    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:37:55.237278 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:37:55.237257    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:38:00.501272 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:00.501238    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7378-8a8456"]
Oct 30 23:38:00.505915 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:00.505896    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7378-8a8456"
Oct 30 23:38:00.661619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:00.661181    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 23:38:00.661619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:00.661281    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 23:38:00.661619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:00.661335    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-qwr4w\""
Oct 30 23:38:00.753704 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:00.745451    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzr48\" (UniqueName: \"kubernetes.io/projected/3268a76b-beac-4b2e-8dd0-990a0f192838-kube-api-access-mzr48\") pod \"test-curl-7378-8a8456\" (UID: \"3268a76b-beac-4b2e-8dd0-990a0f192838\") " pod="opendatahub/test-curl-7378-8a8456"
Oct 30 23:38:00.772198 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:00.772150    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7378-8a8456"]
Oct 30 23:38:00.865633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:00.847289    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzr48\" (UniqueName: \"kubernetes.io/projected/3268a76b-beac-4b2e-8dd0-990a0f192838-kube-api-access-mzr48\") pod \"test-curl-7378-8a8456\" (UID: \"3268a76b-beac-4b2e-8dd0-990a0f192838\") " pod="opendatahub/test-curl-7378-8a8456"
Oct 30 23:38:00.955623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:00.941500    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzr48\" (UniqueName: \"kubernetes.io/projected/3268a76b-beac-4b2e-8dd0-990a0f192838-kube-api-access-mzr48\") pod \"test-curl-7378-8a8456\" (UID: \"3268a76b-beac-4b2e-8dd0-990a0f192838\") " pod="opendatahub/test-curl-7378-8a8456"
Oct 30 23:38:01.121907 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:01.121866    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7378-8a8456"
Oct 30 23:38:01.819305 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:01.819273    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7378-8a8456"]
Oct 30 23:38:01.820091 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:38:01.820062    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3268a76b_beac_4b2e_8dd0_990a0f192838.slice/crio-38061591c3129a993cb4be66963ed397d2a0a5a957a5f1b260c97f02115b7b23 WatchSource:0}: Error finding container 38061591c3129a993cb4be66963ed397d2a0a5a957a5f1b260c97f02115b7b23: Status 404 returned error can't find the container with id 38061591c3129a993cb4be66963ed397d2a0a5a957a5f1b260c97f02115b7b23
Oct 30 23:38:02.290120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:02.290035    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7378-8a8456" event={"ID":"3268a76b-beac-4b2e-8dd0-990a0f192838","Type":"ContainerStarted","Data":"25d6c9d33aeb0a6dd652dfad28f8fdb2749ae8e7d18c9c59c28bad0f315c9fba"}
Oct 30 23:38:02.290120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:02.290083    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7378-8a8456" event={"ID":"3268a76b-beac-4b2e-8dd0-990a0f192838","Type":"ContainerStarted","Data":"38061591c3129a993cb4be66963ed397d2a0a5a957a5f1b260c97f02115b7b23"}
Oct 30 23:38:02.400879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:02.400818    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7378-8a8456" podStartSLOduration=2.400803392 podStartE2EDuration="2.400803392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:38:02.399833116 +0000 UTC m=+1868.287739469" watchObservedRunningTime="2026-10-30 23:38:02.400803392 +0000 UTC m=+1868.288709747"
Oct 30 23:38:02.433394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:02.433366    5078 scope.go:122] "RemoveContainer" containerID="0489e15eee2000515fe03aa275baa11de35bf0ce176888156f7d5307790321f7"
Oct 30 23:38:02.443777 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:02.443758    5078 scope.go:122] "RemoveContainer" containerID="6da0a13fa30a5ef34b885b4ee94f865e191b12b6067ec26d6f861283a95f85ac"
Oct 30 23:38:02.453206 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:02.453177    5078 scope.go:122] "RemoveContainer" containerID="5fb2a6e8c520a4a95a9c26caf7df2f8c02df73fc20713aa48a10261bb8e61c07"
Oct 30 23:38:03.179638 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:03.154993    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"]
Oct 30 23:38:03.348690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:03.348644    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r" podUID="6859c991-1ae2-4055-8d25-5b0d1e858d4a" containerName="istio-proxy" containerID="cri-o://bfbfb43789a1689f912ae423645fa6a2b3a8eecc0e31e46edb01f13459e14de4" gracePeriod=30
Oct 30 23:38:03.463542 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:03.463445    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"]
Oct 30 23:38:03.463830 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:03.463798    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56" podUID="43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" containerName="istio-proxy" containerID="cri-o://d575d4473691f81507b2d29f5c707036259cb7684a80f8ddd69cd80b2507aae3" gracePeriod=30
Oct 30 23:38:03.842169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:03.842102    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7378-8a8456"]
Oct 30 23:38:03.846809 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:03.846779    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7378-8a8456"]
Oct 30 23:38:04.225357 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.225241    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"]
Oct 30 23:38:04.226238 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.226182    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk" podUID="3a1da554-65ce-4df0-a33b-761591100ac2" containerName="istio-proxy" containerID="cri-o://f4b4c3290c428cfe8fa298bf23f342aaa6895d5940cc203f87f2ff02a4cfde55" gracePeriod=30
Oct 30 23:38:04.356067 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.356034    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7378-8a8456" podUID="3268a76b-beac-4b2e-8dd0-990a0f192838" containerName="test-curl-7378-8a8456" containerID="cri-o://25d6c9d33aeb0a6dd652dfad28f8fdb2749ae8e7d18c9c59c28bad0f315c9fba" gracePeriod=2
Oct 30 23:38:04.452745 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:38:04.452536    5078 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7378-8a8456\" is forbidden: User \"system:node:ip-10-0-130-205.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-205.us-west-1.compute.internal' and this object" podUID="3268a76b-beac-4b2e-8dd0-990a0f192838" pod="opendatahub/test-curl-7378-8a8456"
Oct 30 23:38:04.711845 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.711808    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd"]
Oct 30 23:38:04.712472 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.712457    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3268a76b-beac-4b2e-8dd0-990a0f192838" containerName="test-curl-7378-8a8456"
Oct 30 23:38:04.712517 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.712474    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3268a76b-beac-4b2e-8dd0-990a0f192838" containerName="test-curl-7378-8a8456"
Oct 30 23:38:04.712627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.712606    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3268a76b-beac-4b2e-8dd0-990a0f192838" containerName="test-curl-7378-8a8456"
Oct 30 23:38:04.717712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.717694    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd"
Oct 30 23:38:04.768466 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:38:04.768429    5078 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7378-8a8456\" is forbidden: User \"system:node:ip-10-0-130-205.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-205.us-west-1.compute.internal' and this object" podUID="3268a76b-beac-4b2e-8dd0-990a0f192838" pod="opendatahub/test-curl-7378-8a8456"
Oct 30 23:38:04.770210 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.769755    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:38:04.770210 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.769785    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:38:04.770210 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.769866    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:38:04.770210 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.769928    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:38:04.772574 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.772554    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd"]
Oct 30 23:38:04.803105 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.803075    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgd8b\" (UniqueName: \"kubernetes.io/projected/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-kube-api-access-bgd8b\") pod \"maas-api-revoke-keys-e2e-mt-55666687-4qzjd\" (UID: \"d4ccbc2f-d688-471f-bef0-60d3f6bd456b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd"
Oct 30 23:38:04.803217 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.803131    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-55666687-4qzjd\" (UID: \"d4ccbc2f-d688-471f-bef0-60d3f6bd456b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd"
Oct 30 23:38:04.803294 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.803281    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-55666687-4qzjd\" (UID: \"d4ccbc2f-d688-471f-bef0-60d3f6bd456b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd"
Oct 30 23:38:04.904751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.904721    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-55666687-4qzjd\" (UID: \"d4ccbc2f-d688-471f-bef0-60d3f6bd456b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd"
Oct 30 23:38:04.904910 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.904841    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-55666687-4qzjd\" (UID: \"d4ccbc2f-d688-471f-bef0-60d3f6bd456b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd"
Oct 30 23:38:04.904971 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.904959    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgd8b\" (UniqueName: \"kubernetes.io/projected/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-kube-api-access-bgd8b\") pod \"maas-api-revoke-keys-e2e-mt-55666687-4qzjd\" (UID: \"d4ccbc2f-d688-471f-bef0-60d3f6bd456b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd"
Oct 30 23:38:04.905349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.905322    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-55666687-4qzjd\" (UID: \"d4ccbc2f-d688-471f-bef0-60d3f6bd456b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd"
Oct 30 23:38:04.907268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.907246    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-55666687-4qzjd\" (UID: \"d4ccbc2f-d688-471f-bef0-60d3f6bd456b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd"
Oct 30 23:38:04.961309 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:04.961287    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgd8b\" (UniqueName: \"kubernetes.io/projected/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-kube-api-access-bgd8b\") pod \"maas-api-revoke-keys-e2e-mt-55666687-4qzjd\" (UID: \"d4ccbc2f-d688-471f-bef0-60d3f6bd456b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd"
Oct 30 23:38:05.034063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:05.034007    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd"
Oct 30 23:38:05.413990 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:05.413963    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd"]
Oct 30 23:38:05.414701 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:38:05.414671    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ccbc2f_d688_471f_bef0_60d3f6bd456b.slice/crio-2ff015480231468985f8a47b1cb14a4bcdcec372b58a7abc5b9ae49697e2db33 WatchSource:0}: Error finding container 2ff015480231468985f8a47b1cb14a4bcdcec372b58a7abc5b9ae49697e2db33: Status 404 returned error can't find the container with id 2ff015480231468985f8a47b1cb14a4bcdcec372b58a7abc5b9ae49697e2db33
Oct 30 23:38:05.984435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:05.984357    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:38:05.984572 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:05.984426    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r" podUID="6859c991-1ae2-4055-8d25-5b0d1e858d4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:38:06.371701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:06.371674    5078 generic.go:356] "Generic (PLEG): container finished" podID="d4ccbc2f-d688-471f-bef0-60d3f6bd456b" containerID="8d0c2464b7164e633b97d104922e28c90f8c14c7fe9281b4c88991db2aab5b13" exitCode=0
Oct 30 23:38:06.371808 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:06.371758    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd" event={"ID":"d4ccbc2f-d688-471f-bef0-60d3f6bd456b","Type":"ContainerDied","Data":"8d0c2464b7164e633b97d104922e28c90f8c14c7fe9281b4c88991db2aab5b13"}
Oct 30 23:38:06.371855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:06.371813    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd" event={"ID":"d4ccbc2f-d688-471f-bef0-60d3f6bd456b","Type":"ContainerStarted","Data":"2ff015480231468985f8a47b1cb14a4bcdcec372b58a7abc5b9ae49697e2db33"}
Oct 30 23:38:06.613576 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:06.613554    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7378-8a8456"
Oct 30 23:38:06.664349 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:38:06.664316    5078 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7378-8a8456\" is forbidden: User \"system:node:ip-10-0-130-205.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-205.us-west-1.compute.internal' and this object" podUID="3268a76b-beac-4b2e-8dd0-990a0f192838" pod="opendatahub/test-curl-7378-8a8456"
Oct 30 23:38:06.742758 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:06.742733    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3268a76b-beac-4b2e-8dd0-990a0f192838-kube-api-access-mzr48\" (UniqueName: \"kubernetes.io/projected/3268a76b-beac-4b2e-8dd0-990a0f192838-kube-api-access-mzr48\") pod \"3268a76b-beac-4b2e-8dd0-990a0f192838\" (UID: \"3268a76b-beac-4b2e-8dd0-990a0f192838\") "
Oct 30 23:38:06.744965 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:06.744941    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3268a76b-beac-4b2e-8dd0-990a0f192838-kube-api-access-mzr48" pod "3268a76b-beac-4b2e-8dd0-990a0f192838" (UID: "3268a76b-beac-4b2e-8dd0-990a0f192838"). InnerVolumeSpecName "kube-api-access-mzr48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:38:06.844969 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:06.844894    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzr48\" (UniqueName: \"kubernetes.io/projected/3268a76b-beac-4b2e-8dd0-990a0f192838-kube-api-access-mzr48\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:07.380067 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:07.380034    5078 generic.go:356] "Generic (PLEG): container finished" podID="3268a76b-beac-4b2e-8dd0-990a0f192838" containerID="25d6c9d33aeb0a6dd652dfad28f8fdb2749ae8e7d18c9c59c28bad0f315c9fba" exitCode=137
Oct 30 23:38:07.380285 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:07.380082    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7378-8a8456"
Oct 30 23:38:07.380285 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:07.380126    5078 scope.go:122] "RemoveContainer" containerID="25d6c9d33aeb0a6dd652dfad28f8fdb2749ae8e7d18c9c59c28bad0f315c9fba"
Oct 30 23:38:07.396671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:07.393345    5078 scope.go:122] "RemoveContainer" containerID="25d6c9d33aeb0a6dd652dfad28f8fdb2749ae8e7d18c9c59c28bad0f315c9fba"
Oct 30 23:38:07.396892 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:38:07.396868    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d6c9d33aeb0a6dd652dfad28f8fdb2749ae8e7d18c9c59c28bad0f315c9fba\": container with ID starting with 25d6c9d33aeb0a6dd652dfad28f8fdb2749ae8e7d18c9c59c28bad0f315c9fba not found: ID does not exist" containerID="25d6c9d33aeb0a6dd652dfad28f8fdb2749ae8e7d18c9c59c28bad0f315c9fba"
Oct 30 23:38:07.396956 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:07.396901    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d6c9d33aeb0a6dd652dfad28f8fdb2749ae8e7d18c9c59c28bad0f315c9fba"} err="failed to get container status \"25d6c9d33aeb0a6dd652dfad28f8fdb2749ae8e7d18c9c59c28bad0f315c9fba\": rpc error: code = NotFound desc = could not find container \"25d6c9d33aeb0a6dd652dfad28f8fdb2749ae8e7d18c9c59c28bad0f315c9fba\": container with ID starting with 25d6c9d33aeb0a6dd652dfad28f8fdb2749ae8e7d18c9c59c28bad0f315c9fba not found: ID does not exist"
Oct 30 23:38:07.431077 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:38:07.431046    5078 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7378-8a8456\" is forbidden: User \"system:node:ip-10-0-130-205.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-205.us-west-1.compute.internal' and this object" podUID="3268a76b-beac-4b2e-8dd0-990a0f192838" pod="opendatahub/test-curl-7378-8a8456"
Oct 30 23:38:07.481669 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:38:07.481632    5078 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7378-8a8456\" is forbidden: User \"system:node:ip-10-0-130-205.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-205.us-west-1.compute.internal' and this object" podUID="3268a76b-beac-4b2e-8dd0-990a0f192838" pod="opendatahub/test-curl-7378-8a8456"
Oct 30 23:38:07.563935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:07.563909    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd"
Oct 30 23:38:07.614855 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:38:07.614822    5078 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7378-8a8456\" is forbidden: User \"system:node:ip-10-0-130-205.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-205.us-west-1.compute.internal' and this object" podUID="3268a76b-beac-4b2e-8dd0-990a0f192838" pod="opendatahub/test-curl-7378-8a8456"
Oct 30 23:38:07.658486 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:07.658406    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-maas-api-openshift-ca\") pod \"d4ccbc2f-d688-471f-bef0-60d3f6bd456b\" (UID: \"d4ccbc2f-d688-471f-bef0-60d3f6bd456b\") "
Oct 30 23:38:07.658486 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:07.658482    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-kube-api-access-bgd8b\" (UniqueName: \"kubernetes.io/projected/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-kube-api-access-bgd8b\") pod \"d4ccbc2f-d688-471f-bef0-60d3f6bd456b\" (UID: \"d4ccbc2f-d688-471f-bef0-60d3f6bd456b\") "
Oct 30 23:38:07.658711 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:07.658549    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-maas-api-xks-ca\") pod \"d4ccbc2f-d688-471f-bef0-60d3f6bd456b\" (UID: \"d4ccbc2f-d688-471f-bef0-60d3f6bd456b\") "
Oct 30 23:38:07.658828 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:07.658799    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-maas-api-openshift-ca" pod "d4ccbc2f-d688-471f-bef0-60d3f6bd456b" (UID: "d4ccbc2f-d688-471f-bef0-60d3f6bd456b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:38:07.659223 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:07.659202    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:07.660837 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:07.660808    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-maas-api-xks-ca" pod "d4ccbc2f-d688-471f-bef0-60d3f6bd456b" (UID: "d4ccbc2f-d688-471f-bef0-60d3f6bd456b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:38:07.660930 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:07.660860    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-kube-api-access-bgd8b" pod "d4ccbc2f-d688-471f-bef0-60d3f6bd456b" (UID: "d4ccbc2f-d688-471f-bef0-60d3f6bd456b"). InnerVolumeSpecName "kube-api-access-bgd8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:38:07.761092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:07.761059    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgd8b\" (UniqueName: \"kubernetes.io/projected/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-kube-api-access-bgd8b\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:07.761215 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:07.761097    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4ccbc2f-d688-471f-bef0-60d3f6bd456b-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:08.299728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.299699    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3268a76b-beac-4b2e-8dd0-990a0f192838" path="/var/lib/kubelet/pods/3268a76b-beac-4b2e-8dd0-990a0f192838/volumes"
Oct 30 23:38:08.395176 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.395150    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd"
Oct 30 23:38:08.396059 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.396023    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd" event={"ID":"d4ccbc2f-d688-471f-bef0-60d3f6bd456b","Type":"ContainerDied","Data":"2ff015480231468985f8a47b1cb14a4bcdcec372b58a7abc5b9ae49697e2db33"}
Oct 30 23:38:08.396264 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.396109    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ff015480231468985f8a47b1cb14a4bcdcec372b58a7abc5b9ae49697e2db33"
Oct 30 23:38:08.798410 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.798388    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:38:08.910384 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.910349    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-kube-api-access-rghjv\" (UniqueName: \"kubernetes.io/projected/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-kube-api-access-rghjv\") pod \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") "
Oct 30 23:38:08.910511 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.910399    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-envoy\") pod \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") "
Oct 30 23:38:08.910511 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.910434    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-credential-socket\") pod \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") "
Oct 30 23:38:08.910511 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.910485    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-workload-certs\") pod \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") "
Oct 30 23:38:08.910697 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.910527    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-podinfo\") pod \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") "
Oct 30 23:38:08.910697 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.910670    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-data\") pod \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") "
Oct 30 23:38:08.911063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.910805    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istiod-ca-cert\") pod \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") "
Oct 30 23:38:08.911063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.910805    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-workload-certs" pod "43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" (UID: "43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:08.911063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.910829    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-credential-socket" pod "43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" (UID: "43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:08.911063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.910891    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-token\" (UniqueName: \"kubernetes.io/projected/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-token\") pod \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") "
Oct 30 23:38:08.911063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.911029    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-data" pod "43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" (UID: "43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:08.911063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.911047    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-workload-socket\") pod \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\" (UID: \"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3\") "
Oct 30 23:38:08.911460 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.911128    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istiod-ca-cert" pod "43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" (UID: "43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:38:08.911460 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.911410    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-workload-socket" pod "43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" (UID: "43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:08.912061 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.912026    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:08.912179 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.912106    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:08.912179 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.912129    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:08.912179 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.912147    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:08.912179 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.912163    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:08.912992 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.912969    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-envoy" pod "43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" (UID: "43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:08.914501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.914448    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-podinfo" pod "43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" (UID: "43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:38:08.914767 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.914746    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-token" pod "43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" (UID: "43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:38:08.914838 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:08.914790    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-kube-api-access-rghjv" pod "43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" (UID: "43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3"). InnerVolumeSpecName "kube-api-access-rghjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:38:09.015055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.015013    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rghjv\" (UniqueName: \"kubernetes.io/projected/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-kube-api-access-rghjv\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.015055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.015059    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.015263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.015075    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.015263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.015091    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.057561 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.057531    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:38:09.121365 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.121330    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:38:09.121534 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.121388    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk" podUID="3a1da554-65ce-4df0-a33b-761591100ac2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:38:09.222146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.222059    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-workload-socket\") pod \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") "
Oct 30 23:38:09.222307 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.222191    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-podinfo\") pod \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") "
Oct 30 23:38:09.222307 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.222232    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-data\") pod \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") "
Oct 30 23:38:09.222429 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.222349    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6859c991-1ae2-4055-8d25-5b0d1e858d4a-kube-api-access-tkhz4\" (UniqueName: \"kubernetes.io/projected/6859c991-1ae2-4055-8d25-5b0d1e858d4a-kube-api-access-tkhz4\") pod \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") "
Oct 30 23:38:09.222429 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.222350    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-workload-socket" pod "6859c991-1ae2-4055-8d25-5b0d1e858d4a" (UID: "6859c991-1ae2-4055-8d25-5b0d1e858d4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:09.222429 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.222392    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-credential-socket\") pod \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") "
Oct 30 23:38:09.222587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.222468    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-token\" (UniqueName: \"kubernetes.io/projected/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-token\") pod \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") "
Oct 30 23:38:09.222587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.222510    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-workload-certs\") pod \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") "
Oct 30 23:38:09.222723 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.222667    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-data" pod "6859c991-1ae2-4055-8d25-5b0d1e858d4a" (UID: "6859c991-1ae2-4055-8d25-5b0d1e858d4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:09.222723 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.222691    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-envoy\") pod \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") "
Oct 30 23:38:09.222843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.222729    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istiod-ca-cert\") pod \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\" (UID: \"6859c991-1ae2-4055-8d25-5b0d1e858d4a\") "
Oct 30 23:38:09.223760 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.223732    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.223890 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.223766    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.223890 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.223868    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-workload-certs" pod "6859c991-1ae2-4055-8d25-5b0d1e858d4a" (UID: "6859c991-1ae2-4055-8d25-5b0d1e858d4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:09.224011 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.223906    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istiod-ca-cert" pod "6859c991-1ae2-4055-8d25-5b0d1e858d4a" (UID: "6859c991-1ae2-4055-8d25-5b0d1e858d4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:38:09.224210 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.224181    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-credential-socket" pod "6859c991-1ae2-4055-8d25-5b0d1e858d4a" (UID: "6859c991-1ae2-4055-8d25-5b0d1e858d4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:09.224866 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.224839    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-podinfo" pod "6859c991-1ae2-4055-8d25-5b0d1e858d4a" (UID: "6859c991-1ae2-4055-8d25-5b0d1e858d4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:38:09.224961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.224907    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6859c991-1ae2-4055-8d25-5b0d1e858d4a-kube-api-access-tkhz4" pod "6859c991-1ae2-4055-8d25-5b0d1e858d4a" (UID: "6859c991-1ae2-4055-8d25-5b0d1e858d4a"). InnerVolumeSpecName "kube-api-access-tkhz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:38:09.225153 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.225138    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-token" pod "6859c991-1ae2-4055-8d25-5b0d1e858d4a" (UID: "6859c991-1ae2-4055-8d25-5b0d1e858d4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:38:09.225756 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.225734    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-envoy" pod "6859c991-1ae2-4055-8d25-5b0d1e858d4a" (UID: "6859c991-1ae2-4055-8d25-5b0d1e858d4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:09.327120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.325915    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.327120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.325951    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkhz4\" (UniqueName: \"kubernetes.io/projected/6859c991-1ae2-4055-8d25-5b0d1e858d4a-kube-api-access-tkhz4\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.327120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.325968    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.327120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.325983    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.327120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.325996    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.327120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.326009    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.327120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.326022    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6859c991-1ae2-4055-8d25-5b0d1e858d4a-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.418653 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.418577    5078 generic.go:356] "Generic (PLEG): container finished" podID="43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" containerID="d575d4473691f81507b2d29f5c707036259cb7684a80f8ddd69cd80b2507aae3" exitCode=0
Oct 30 23:38:09.418843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.418705    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"
Oct 30 23:38:09.418843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.418754    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56" event={"ID":"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3","Type":"ContainerDied","Data":"d575d4473691f81507b2d29f5c707036259cb7684a80f8ddd69cd80b2507aae3"}
Oct 30 23:38:09.418843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.418805    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56" event={"ID":"43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3","Type":"ContainerDied","Data":"fe6ade576114b1979c24de3e169d846a4772a52b307bb79b1574d7f66e303c27"}
Oct 30 23:38:09.418843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.418833    5078 scope.go:122] "RemoveContainer" containerID="d575d4473691f81507b2d29f5c707036259cb7684a80f8ddd69cd80b2507aae3"
Oct 30 23:38:09.421919 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.421895    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk" event={"ID":"3a1da554-65ce-4df0-a33b-761591100ac2","Type":"ContainerDied","Data":"f4b4c3290c428cfe8fa298bf23f342aaa6895d5940cc203f87f2ff02a4cfde55"}
Oct 30 23:38:09.422117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.422091    5078 generic.go:356] "Generic (PLEG): container finished" podID="3a1da554-65ce-4df0-a33b-761591100ac2" containerID="f4b4c3290c428cfe8fa298bf23f342aaa6895d5940cc203f87f2ff02a4cfde55" exitCode=0
Oct 30 23:38:09.434453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.429330    5078 generic.go:356] "Generic (PLEG): container finished" podID="6859c991-1ae2-4055-8d25-5b0d1e858d4a" containerID="bfbfb43789a1689f912ae423645fa6a2b3a8eecc0e31e46edb01f13459e14de4" exitCode=0
Oct 30 23:38:09.434453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.429443    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r" event={"ID":"6859c991-1ae2-4055-8d25-5b0d1e858d4a","Type":"ContainerDied","Data":"bfbfb43789a1689f912ae423645fa6a2b3a8eecc0e31e46edb01f13459e14de4"}
Oct 30 23:38:09.434453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.429481    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r" event={"ID":"6859c991-1ae2-4055-8d25-5b0d1e858d4a","Type":"ContainerDied","Data":"82997942111b3b9c601b9e3bd1998760548b74fb7063158e764c25b289fdf414"}
Oct 30 23:38:09.434453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.429576    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"
Oct 30 23:38:09.436790 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.436721    5078 scope.go:122] "RemoveContainer" containerID="d575d4473691f81507b2d29f5c707036259cb7684a80f8ddd69cd80b2507aae3"
Oct 30 23:38:09.437791 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:38:09.437168    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d575d4473691f81507b2d29f5c707036259cb7684a80f8ddd69cd80b2507aae3\": container with ID starting with d575d4473691f81507b2d29f5c707036259cb7684a80f8ddd69cd80b2507aae3 not found: ID does not exist" containerID="d575d4473691f81507b2d29f5c707036259cb7684a80f8ddd69cd80b2507aae3"
Oct 30 23:38:09.437791 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.437206    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d575d4473691f81507b2d29f5c707036259cb7684a80f8ddd69cd80b2507aae3"} err="failed to get container status \"d575d4473691f81507b2d29f5c707036259cb7684a80f8ddd69cd80b2507aae3\": rpc error: code = NotFound desc = could not find container \"d575d4473691f81507b2d29f5c707036259cb7684a80f8ddd69cd80b2507aae3\": container with ID starting with d575d4473691f81507b2d29f5c707036259cb7684a80f8ddd69cd80b2507aae3 not found: ID does not exist"
Oct 30 23:38:09.437791 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.437231    5078 scope.go:122] "RemoveContainer" containerID="bfbfb43789a1689f912ae423645fa6a2b3a8eecc0e31e46edb01f13459e14de4"
Oct 30 23:38:09.449951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.449934    5078 scope.go:122] "RemoveContainer" containerID="bfbfb43789a1689f912ae423645fa6a2b3a8eecc0e31e46edb01f13459e14de4"
Oct 30 23:38:09.450268 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:38:09.450247    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfbfb43789a1689f912ae423645fa6a2b3a8eecc0e31e46edb01f13459e14de4\": container with ID starting with bfbfb43789a1689f912ae423645fa6a2b3a8eecc0e31e46edb01f13459e14de4 not found: ID does not exist" containerID="bfbfb43789a1689f912ae423645fa6a2b3a8eecc0e31e46edb01f13459e14de4"
Oct 30 23:38:09.450389 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.450273    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfbfb43789a1689f912ae423645fa6a2b3a8eecc0e31e46edb01f13459e14de4"} err="failed to get container status \"bfbfb43789a1689f912ae423645fa6a2b3a8eecc0e31e46edb01f13459e14de4\": rpc error: code = NotFound desc = could not find container \"bfbfb43789a1689f912ae423645fa6a2b3a8eecc0e31e46edb01f13459e14de4\": container with ID starting with bfbfb43789a1689f912ae423645fa6a2b3a8eecc0e31e46edb01f13459e14de4 not found: ID does not exist"
Oct 30 23:38:09.487974 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.487921    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:38:09.667620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.656415    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1da554-65ce-4df0-a33b-761591100ac2-istio-token\" (UniqueName: \"kubernetes.io/projected/3a1da554-65ce-4df0-a33b-761591100ac2-istio-token\") pod \"3a1da554-65ce-4df0-a33b-761591100ac2\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") "
Oct 30 23:38:09.667620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.656481    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a1da554-65ce-4df0-a33b-761591100ac2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1da554-65ce-4df0-a33b-761591100ac2-istio-podinfo\") pod \"3a1da554-65ce-4df0-a33b-761591100ac2\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") "
Oct 30 23:38:09.667620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.656536    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-workload-certs\") pod \"3a1da554-65ce-4df0-a33b-761591100ac2\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") "
Oct 30 23:38:09.667620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.656728    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-istio-data\") pod \"3a1da554-65ce-4df0-a33b-761591100ac2\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") "
Oct 30 23:38:09.667620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.656768    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-credential-socket\") pod \"3a1da554-65ce-4df0-a33b-761591100ac2\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") "
Oct 30 23:38:09.667620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.656813    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a1da554-65ce-4df0-a33b-761591100ac2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1da554-65ce-4df0-a33b-761591100ac2-istiod-ca-cert\") pod \"3a1da554-65ce-4df0-a33b-761591100ac2\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") "
Oct 30 23:38:09.667620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.656875    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-istio-envoy\") pod \"3a1da554-65ce-4df0-a33b-761591100ac2\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") "
Oct 30 23:38:09.667620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.656959    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1da554-65ce-4df0-a33b-761591100ac2-kube-api-access-x68kc\" (UniqueName: \"kubernetes.io/projected/3a1da554-65ce-4df0-a33b-761591100ac2-kube-api-access-x68kc\") pod \"3a1da554-65ce-4df0-a33b-761591100ac2\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") "
Oct 30 23:38:09.667620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.657090    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-workload-socket\") pod \"3a1da554-65ce-4df0-a33b-761591100ac2\" (UID: \"3a1da554-65ce-4df0-a33b-761591100ac2\") "
Oct 30 23:38:09.679016 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.675857    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-credential-socket" pod "3a1da554-65ce-4df0-a33b-761591100ac2" (UID: "3a1da554-65ce-4df0-a33b-761591100ac2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:09.679016 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.676048    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-workload-certs" pod "3a1da554-65ce-4df0-a33b-761591100ac2" (UID: "3a1da554-65ce-4df0-a33b-761591100ac2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:09.679016 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.676203    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-istio-data" pod "3a1da554-65ce-4df0-a33b-761591100ac2" (UID: "3a1da554-65ce-4df0-a33b-761591100ac2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:09.679016 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.678529    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1da554-65ce-4df0-a33b-761591100ac2-istio-token" pod "3a1da554-65ce-4df0-a33b-761591100ac2" (UID: "3a1da554-65ce-4df0-a33b-761591100ac2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:38:09.688716 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.680053    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1da554-65ce-4df0-a33b-761591100ac2-istiod-ca-cert" pod "3a1da554-65ce-4df0-a33b-761591100ac2" (UID: "3a1da554-65ce-4df0-a33b-761591100ac2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:38:09.688716 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.680416    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-workload-socket" pod "3a1da554-65ce-4df0-a33b-761591100ac2" (UID: "3a1da554-65ce-4df0-a33b-761591100ac2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:09.725621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.717880    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"]
Oct 30 23:38:09.731688 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.731068    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-istio-envoy" pod "3a1da554-65ce-4df0-a33b-761591100ac2" (UID: "3a1da554-65ce-4df0-a33b-761591100ac2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:38:09.731859 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.731704    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a1da554-65ce-4df0-a33b-761591100ac2-istio-podinfo" pod "3a1da554-65ce-4df0-a33b-761591100ac2" (UID: "3a1da554-65ce-4df0-a33b-761591100ac2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:38:09.735689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.734284    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1da554-65ce-4df0-a33b-761591100ac2-kube-api-access-x68kc" pod "3a1da554-65ce-4df0-a33b-761591100ac2" (UID: "3a1da554-65ce-4df0-a33b-761591100ac2"). InnerVolumeSpecName "kube-api-access-x68kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:38:09.736018 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.735999    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-czg56"]
Oct 30 23:38:09.744476 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.742994    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"]
Oct 30 23:38:09.748934 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.748905    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-p494r"]
Oct 30 23:38:09.760121 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.760087    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.760121 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.760111    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a1da554-65ce-4df0-a33b-761591100ac2-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.760121 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.760121    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a1da554-65ce-4df0-a33b-761591100ac2-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.760366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.760131    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.760366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.760139    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.760366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.760147    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.760366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.760156    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a1da554-65ce-4df0-a33b-761591100ac2-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.760366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.760163    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a1da554-65ce-4df0-a33b-761591100ac2-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:09.760366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:09.760171    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x68kc\" (UniqueName: \"kubernetes.io/projected/3a1da554-65ce-4df0-a33b-761591100ac2-kube-api-access-x68kc\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:10.300367 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:10.300334    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" path="/var/lib/kubelet/pods/43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3/volumes"
Oct 30 23:38:10.300922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:10.300906    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6859c991-1ae2-4055-8d25-5b0d1e858d4a" path="/var/lib/kubelet/pods/6859c991-1ae2-4055-8d25-5b0d1e858d4a/volumes"
Oct 30 23:38:10.438853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:10.438819    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"
Oct 30 23:38:10.438853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:10.438831    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk" event={"ID":"3a1da554-65ce-4df0-a33b-761591100ac2","Type":"ContainerDied","Data":"9de1ee282b2dac070b0da864ba4193494e418d2fd253aa7be4e259eb199bb3d7"}
Oct 30 23:38:10.439110 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:10.438881    5078 scope.go:122] "RemoveContainer" containerID="f4b4c3290c428cfe8fa298bf23f342aaa6895d5940cc203f87f2ff02a4cfde55"
Oct 30 23:38:10.605042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:10.604965    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"]
Oct 30 23:38:10.653983 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:10.653955    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-7wwxk"]
Oct 30 23:38:12.299666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:12.299635    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a1da554-65ce-4df0-a33b-761591100ac2" path="/var/lib/kubelet/pods/3a1da554-65ce-4df0-a33b-761591100ac2/volumes"
Oct 30 23:38:16.211442 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:16.211407    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml"]
Oct 30 23:38:16.332929 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:16.332879    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v"]
Oct 30 23:38:17.592850 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:38:17.592812    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:38:21.211981 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:21.211923    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml" podUID="92708dca-2c92-4ca1-b91d-57c81b176c79" containerName="payload-processing" containerID="cri-o://57a138c7819944198154c009d090014e5a858cd6220c8ad63075e44405dae07f" gracePeriod=25
Oct 30 23:38:21.334144 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:21.334104    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v" podUID="53db4c77-5707-4260-8d1b-83d85e0d9bc1" containerName="payload-pre-processing" containerID="cri-o://4846ea023d1a1046038706f2c241942ea0829375ee13b369db54be8704caf7c7" gracePeriod=25
Oct 30 23:38:24.444687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:24.444640    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml" podUID="92708dca-2c92-4ca1-b91d-57c81b176c79" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:38:24.445118 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:24.444725    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v" podUID="53db4c77-5707-4260-8d1b-83d85e0d9bc1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:38:34.444485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:34.444397    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml" podUID="92708dca-2c92-4ca1-b91d-57c81b176c79" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:38:34.444485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:34.444397    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v" podUID="53db4c77-5707-4260-8d1b-83d85e0d9bc1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:38:41.492525 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.492499    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml"
Oct 30 23:38:41.562314 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.562284    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92708dca-2c92-4ca1-b91d-57c81b176c79-kube-api-access-zjjkc\" (UniqueName: \"kubernetes.io/projected/92708dca-2c92-4ca1-b91d-57c81b176c79-kube-api-access-zjjkc\") pod \"92708dca-2c92-4ca1-b91d-57c81b176c79\" (UID: \"92708dca-2c92-4ca1-b91d-57c81b176c79\") "
Oct 30 23:38:41.562704 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.562687    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92708dca-2c92-4ca1-b91d-57c81b176c79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92708dca-2c92-4ca1-b91d-57c81b176c79-plugins-config-volume\") pod \"92708dca-2c92-4ca1-b91d-57c81b176c79\" (UID: \"92708dca-2c92-4ca1-b91d-57c81b176c79\") "
Oct 30 23:38:41.563064 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.563037    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92708dca-2c92-4ca1-b91d-57c81b176c79-plugins-config-volume" pod "92708dca-2c92-4ca1-b91d-57c81b176c79" (UID: "92708dca-2c92-4ca1-b91d-57c81b176c79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:38:41.563630 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.563609    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92708dca-2c92-4ca1-b91d-57c81b176c79-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:41.566572 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.566540    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92708dca-2c92-4ca1-b91d-57c81b176c79-kube-api-access-zjjkc" pod "92708dca-2c92-4ca1-b91d-57c81b176c79" (UID: "92708dca-2c92-4ca1-b91d-57c81b176c79"). InnerVolumeSpecName "kube-api-access-zjjkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:38:41.606111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.606085    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v"
Oct 30 23:38:41.664610 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.664556    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53db4c77-5707-4260-8d1b-83d85e0d9bc1-kube-api-access-kj64g\" (UniqueName: \"kubernetes.io/projected/53db4c77-5707-4260-8d1b-83d85e0d9bc1-kube-api-access-kj64g\") pod \"53db4c77-5707-4260-8d1b-83d85e0d9bc1\" (UID: \"53db4c77-5707-4260-8d1b-83d85e0d9bc1\") "
Oct 30 23:38:41.664889 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.664868    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53db4c77-5707-4260-8d1b-83d85e0d9bc1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53db4c77-5707-4260-8d1b-83d85e0d9bc1-plugins-config-volume\") pod \"53db4c77-5707-4260-8d1b-83d85e0d9bc1\" (UID: \"53db4c77-5707-4260-8d1b-83d85e0d9bc1\") "
Oct 30 23:38:41.665278 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.665253    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53db4c77-5707-4260-8d1b-83d85e0d9bc1-plugins-config-volume" pod "53db4c77-5707-4260-8d1b-83d85e0d9bc1" (UID: "53db4c77-5707-4260-8d1b-83d85e0d9bc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:38:41.665468 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.665452    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjjkc\" (UniqueName: \"kubernetes.io/projected/92708dca-2c92-4ca1-b91d-57c81b176c79-kube-api-access-zjjkc\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:41.665532 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.665475    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53db4c77-5707-4260-8d1b-83d85e0d9bc1-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:41.667004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.666982    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53db4c77-5707-4260-8d1b-83d85e0d9bc1-kube-api-access-kj64g" pod "53db4c77-5707-4260-8d1b-83d85e0d9bc1" (UID: "53db4c77-5707-4260-8d1b-83d85e0d9bc1"). InnerVolumeSpecName "kube-api-access-kj64g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:38:41.701303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.701272    5078 generic.go:356] "Generic (PLEG): container finished" podID="92708dca-2c92-4ca1-b91d-57c81b176c79" containerID="57a138c7819944198154c009d090014e5a858cd6220c8ad63075e44405dae07f" exitCode=0
Oct 30 23:38:41.701448 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.701320    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml"
Oct 30 23:38:41.701448 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.701361    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml" event={"ID":"92708dca-2c92-4ca1-b91d-57c81b176c79","Type":"ContainerDied","Data":"57a138c7819944198154c009d090014e5a858cd6220c8ad63075e44405dae07f"}
Oct 30 23:38:41.701448 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.701417    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml" event={"ID":"92708dca-2c92-4ca1-b91d-57c81b176c79","Type":"ContainerDied","Data":"d0cd04cb8e1a206ff53b2cc5266b8b274b80439a6e0aa62cb57bc71440af8f17"}
Oct 30 23:38:41.701448 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.701445    5078 scope.go:122] "RemoveContainer" containerID="57a138c7819944198154c009d090014e5a858cd6220c8ad63075e44405dae07f"
Oct 30 23:38:41.702839 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.702815    5078 generic.go:356] "Generic (PLEG): container finished" podID="53db4c77-5707-4260-8d1b-83d85e0d9bc1" containerID="4846ea023d1a1046038706f2c241942ea0829375ee13b369db54be8704caf7c7" exitCode=0
Oct 30 23:38:41.703136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.702842    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v" event={"ID":"53db4c77-5707-4260-8d1b-83d85e0d9bc1","Type":"ContainerDied","Data":"4846ea023d1a1046038706f2c241942ea0829375ee13b369db54be8704caf7c7"}
Oct 30 23:38:41.703136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.702863    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v"
Oct 30 23:38:41.703136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.702890    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v" event={"ID":"53db4c77-5707-4260-8d1b-83d85e0d9bc1","Type":"ContainerDied","Data":"75f65366d100f335e388044a0ec73efb7c491dcf58517538915cea32ba7ca169"}
Oct 30 23:38:41.713115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.713097    5078 scope.go:122] "RemoveContainer" containerID="57a138c7819944198154c009d090014e5a858cd6220c8ad63075e44405dae07f"
Oct 30 23:38:41.713462 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:38:41.713436    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57a138c7819944198154c009d090014e5a858cd6220c8ad63075e44405dae07f\": container with ID starting with 57a138c7819944198154c009d090014e5a858cd6220c8ad63075e44405dae07f not found: ID does not exist" containerID="57a138c7819944198154c009d090014e5a858cd6220c8ad63075e44405dae07f"
Oct 30 23:38:41.713569 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.713470    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57a138c7819944198154c009d090014e5a858cd6220c8ad63075e44405dae07f"} err="failed to get container status \"57a138c7819944198154c009d090014e5a858cd6220c8ad63075e44405dae07f\": rpc error: code = NotFound desc = could not find container \"57a138c7819944198154c009d090014e5a858cd6220c8ad63075e44405dae07f\": container with ID starting with 57a138c7819944198154c009d090014e5a858cd6220c8ad63075e44405dae07f not found: ID does not exist"
Oct 30 23:38:41.713569 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.713489    5078 scope.go:122] "RemoveContainer" containerID="4846ea023d1a1046038706f2c241942ea0829375ee13b369db54be8704caf7c7"
Oct 30 23:38:41.722516 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.722501    5078 scope.go:122] "RemoveContainer" containerID="4846ea023d1a1046038706f2c241942ea0829375ee13b369db54be8704caf7c7"
Oct 30 23:38:41.722823 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:38:41.722806    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4846ea023d1a1046038706f2c241942ea0829375ee13b369db54be8704caf7c7\": container with ID starting with 4846ea023d1a1046038706f2c241942ea0829375ee13b369db54be8704caf7c7 not found: ID does not exist" containerID="4846ea023d1a1046038706f2c241942ea0829375ee13b369db54be8704caf7c7"
Oct 30 23:38:41.722879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.722832    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4846ea023d1a1046038706f2c241942ea0829375ee13b369db54be8704caf7c7"} err="failed to get container status \"4846ea023d1a1046038706f2c241942ea0829375ee13b369db54be8704caf7c7\": rpc error: code = NotFound desc = could not find container \"4846ea023d1a1046038706f2c241942ea0829375ee13b369db54be8704caf7c7\": container with ID starting with 4846ea023d1a1046038706f2c241942ea0829375ee13b369db54be8704caf7c7 not found: ID does not exist"
Oct 30 23:38:41.766425 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.766395    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj64g\" (UniqueName: \"kubernetes.io/projected/53db4c77-5707-4260-8d1b-83d85e0d9bc1-kube-api-access-kj64g\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:38:41.843143 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.843112    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml"]
Oct 30 23:38:41.848527 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.848497    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-2tdml"]
Oct 30 23:38:41.951843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.951699    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v"]
Oct 30 23:38:41.955465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:41.955438    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-55666687-84b5d7b947-l2m5v"]
Oct 30 23:38:42.299041 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:42.299011    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53db4c77-5707-4260-8d1b-83d85e0d9bc1" path="/var/lib/kubelet/pods/53db4c77-5707-4260-8d1b-83d85e0d9bc1/volumes"
Oct 30 23:38:42.299422 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:38:42.299409    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92708dca-2c92-4ca1-b91d-57c81b176c79" path="/var/lib/kubelet/pods/92708dca-2c92-4ca1-b91d-57c81b176c79/volumes"
Oct 30 23:38:47.592735 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:38:47.592692    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:39:00.935651 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:00.935590    5078 generic.go:356] "Generic (PLEG): container finished" podID="eb7f4356-b694-4ebb-a65e-8f7782591aea" containerID="f430980ddda5807584ae8cf577187e1ee40beb081b49c4c82c969495242684b9" exitCode=0
Oct 30 23:39:00.936098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:00.935664    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4" event={"ID":"eb7f4356-b694-4ebb-a65e-8f7782591aea","Type":"ContainerDied","Data":"f430980ddda5807584ae8cf577187e1ee40beb081b49c4c82c969495242684b9"}
Oct 30 23:39:00.936162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:00.936123    5078 scope.go:122] "RemoveContainer" containerID="f430980ddda5807584ae8cf577187e1ee40beb081b49c4c82c969495242684b9"
Oct 30 23:39:00.947821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:00.947795    5078 generic.go:356] "Generic (PLEG): container finished" podID="c8b4ee1d-90dd-416a-9955-cc54ad064723" containerID="12b24b5e11c1cd12348c0a391bd1e210ffc895b878b5be170495658ea95ddb13" exitCode=0
Oct 30 23:39:00.947887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:00.947853    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c" event={"ID":"c8b4ee1d-90dd-416a-9955-cc54ad064723","Type":"ContainerDied","Data":"12b24b5e11c1cd12348c0a391bd1e210ffc895b878b5be170495658ea95ddb13"}
Oct 30 23:39:00.948303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:00.948282    5078 scope.go:122] "RemoveContainer" containerID="12b24b5e11c1cd12348c0a391bd1e210ffc895b878b5be170495658ea95ddb13"
Oct 30 23:39:00.948785 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:00.948750    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:39:00.948871 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:00.948840    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:39:00.948967 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:00.948890    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:39:00.949898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:00.949873    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:39:00.949997 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:00.949946    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:39:00.949997 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:00.949978    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:39:01.982622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:01.981732    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c" event={"ID":"c8b4ee1d-90dd-416a-9955-cc54ad064723","Type":"ContainerStarted","Data":"2d792d082bc075db64d00d86bd328798a1a312e75575061d5a9f48639fc4681c"}
Oct 30 23:39:01.986617 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:01.984858    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4" event={"ID":"eb7f4356-b694-4ebb-a65e-8f7782591aea","Type":"ContainerStarted","Data":"0892ed9f3ad748423685174496cd366240b726ecf65c98508490c3d22111b935"}
Oct 30 23:39:02.146322 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:02.146292    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:39:02.421316 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:02.421290    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:39:02.422192 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:02.422169    5078 patch_prober.go:29] interesting pod/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 23:39:02.422277 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:02.422221    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4" podUID="eb7f4356-b694-4ebb-a65e-8f7782591aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 23:39:02.511843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:02.511800    5078 scope.go:122] "RemoveContainer" containerID="f2b15d9131f271ddab270ed40e9a96b7ad979b8cffb4ed7fe6fbe2518190524c"
Oct 30 23:39:02.523420 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:02.523398    5078 scope.go:122] "RemoveContainer" containerID="c69ed29a03136a5424d5394ef0bce65767931efec76d4685dd8b7d8ae8e1e7a9"
Oct 30 23:39:02.533098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:02.533078    5078 scope.go:122] "RemoveContainer" containerID="93d71478d09e496ff215f600d545869e458c38c6520bd5ec5860935af20b8398"
Oct 30 23:39:02.923116 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:02.923085    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:39:02.924284 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:02.924260    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 23:39:02.924427 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:02.924304    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c" podUID="c8b4ee1d-90dd-416a-9955-cc54ad064723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 23:39:03.116963 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.116925    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx"]
Oct 30 23:39:03.117800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.117778    5078 cpu_manager.go:414] "containerMap: removing container" podUID="92708dca-2c92-4ca1-b91d-57c81b176c79" containerName="payload-processing"
Oct 30 23:39:03.117800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.117799    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92708dca-2c92-4ca1-b91d-57c81b176c79" containerName="payload-processing"
Oct 30 23:39:03.118111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.117814    5078 cpu_manager.go:414] "containerMap: removing container" podUID="43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" containerName="istio-proxy"
Oct 30 23:39:03.118111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.117823    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" containerName="istio-proxy"
Oct 30 23:39:03.118111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.117835    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3a1da554-65ce-4df0-a33b-761591100ac2" containerName="istio-proxy"
Oct 30 23:39:03.118111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.117841    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1da554-65ce-4df0-a33b-761591100ac2" containerName="istio-proxy"
Oct 30 23:39:03.118111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.117874    5078 cpu_manager.go:414] "containerMap: removing container" podUID="6859c991-1ae2-4055-8d25-5b0d1e858d4a" containerName="istio-proxy"
Oct 30 23:39:03.118111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.117881    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6859c991-1ae2-4055-8d25-5b0d1e858d4a" containerName="istio-proxy"
Oct 30 23:39:03.118111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.117907    5078 cpu_manager.go:414] "containerMap: removing container" podUID="53db4c77-5707-4260-8d1b-83d85e0d9bc1" containerName="payload-pre-processing"
Oct 30 23:39:03.118111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.117914    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53db4c77-5707-4260-8d1b-83d85e0d9bc1" containerName="payload-pre-processing"
Oct 30 23:39:03.118111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.117935    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d4ccbc2f-d688-471f-bef0-60d3f6bd456b" containerName="revoke-keys"
Oct 30 23:39:03.118111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.117942    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4ccbc2f-d688-471f-bef0-60d3f6bd456b" containerName="revoke-keys"
Oct 30 23:39:03.118111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.118099    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="6859c991-1ae2-4055-8d25-5b0d1e858d4a" containerName="istio-proxy"
Oct 30 23:39:03.118111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.118116    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="53db4c77-5707-4260-8d1b-83d85e0d9bc1" containerName="payload-pre-processing"
Oct 30 23:39:03.118772 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.118131    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4ccbc2f-d688-471f-bef0-60d3f6bd456b" containerName="revoke-keys"
Oct 30 23:39:03.118772 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.118145    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="92708dca-2c92-4ca1-b91d-57c81b176c79" containerName="payload-processing"
Oct 30 23:39:03.118772 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.118157    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d12dbb-7f9a-483f-b1f9-3eeca3e2f3d3" containerName="istio-proxy"
Oct 30 23:39:03.118772 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.118166    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1da554-65ce-4df0-a33b-761591100ac2" containerName="istio-proxy"
Oct 30 23:39:03.124339 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.124318    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx"
Oct 30 23:39:03.138485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.138462    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472179d6-45c3-4598-8961-23c2f46b6fea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-721fec8b-nmplx\" (UID: \"472179d6-45c3-4598-8961-23c2f46b6fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx"
Oct 30 23:39:03.138715 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.138698    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472179d6-45c3-4598-8961-23c2f46b6fea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-721fec8b-nmplx\" (UID: \"472179d6-45c3-4598-8961-23c2f46b6fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx"
Oct 30 23:39:03.138804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.138757    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5647\" (UniqueName: \"kubernetes.io/projected/472179d6-45c3-4598-8961-23c2f46b6fea-kube-api-access-d5647\") pod \"maas-api-revoke-sub-721fec8b-nmplx\" (UID: \"472179d6-45c3-4598-8961-23c2f46b6fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx"
Oct 30 23:39:03.184213 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.184183    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:39:03.185268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.184526    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:39:03.185417 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.184616    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:39:03.185511 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.184660    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:39:03.214691 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.213201    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx"]
Oct 30 23:39:03.239939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.239909    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472179d6-45c3-4598-8961-23c2f46b6fea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-721fec8b-nmplx\" (UID: \"472179d6-45c3-4598-8961-23c2f46b6fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx"
Oct 30 23:39:03.240124 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.240102    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472179d6-45c3-4598-8961-23c2f46b6fea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-721fec8b-nmplx\" (UID: \"472179d6-45c3-4598-8961-23c2f46b6fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx"
Oct 30 23:39:03.240196 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.240166    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5647\" (UniqueName: \"kubernetes.io/projected/472179d6-45c3-4598-8961-23c2f46b6fea-kube-api-access-d5647\") pod \"maas-api-revoke-sub-721fec8b-nmplx\" (UID: \"472179d6-45c3-4598-8961-23c2f46b6fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx"
Oct 30 23:39:03.240949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.240928    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472179d6-45c3-4598-8961-23c2f46b6fea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-721fec8b-nmplx\" (UID: \"472179d6-45c3-4598-8961-23c2f46b6fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx"
Oct 30 23:39:03.243001 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.242983    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472179d6-45c3-4598-8961-23c2f46b6fea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-721fec8b-nmplx\" (UID: \"472179d6-45c3-4598-8961-23c2f46b6fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx"
Oct 30 23:39:03.316052 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.316017    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5647\" (UniqueName: \"kubernetes.io/projected/472179d6-45c3-4598-8961-23c2f46b6fea-kube-api-access-d5647\") pod \"maas-api-revoke-sub-721fec8b-nmplx\" (UID: \"472179d6-45c3-4598-8961-23c2f46b6fea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx"
Oct 30 23:39:03.423815 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.423729    5078 patch_prober.go:29] interesting pod/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 23:39:03.423815 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.423789    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4" podUID="eb7f4356-b694-4ebb-a65e-8f7782591aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 23:39:03.438340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.438314    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx"
Oct 30 23:39:03.897623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.897316    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx"]
Oct 30 23:39:03.928227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.926998    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 23:39:03.928227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:03.927069    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c" podUID="c8b4ee1d-90dd-416a-9955-cc54ad064723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 23:39:04.091622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:04.085027    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx" event={"ID":"472179d6-45c3-4598-8961-23c2f46b6fea","Type":"ContainerStarted","Data":"a353c5ee00e0cc47659a5079100a26b3f39348c784784a45f0ff5a14227a83d1"}
Oct 30 23:39:04.421008 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:04.420977    5078 patch_prober.go:29] interesting pod/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 23:39:04.421380 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:04.421033    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4" podUID="eb7f4356-b694-4ebb-a65e-8f7782591aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 23:39:04.923633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:04.923056    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 23:39:04.923633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:04.923128    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c" podUID="c8b4ee1d-90dd-416a-9955-cc54ad064723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 23:39:05.079621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:05.071868    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:39:05.113464 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:05.113426    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx" event={"ID":"472179d6-45c3-4598-8961-23c2f46b6fea","Type":"ContainerStarted","Data":"262afe6c9f0a38e124ba0fed3e5d8e9d711f4392bb9ba2064a5cbc7ed416f5d6"}
Oct 30 23:39:05.243111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:05.242981    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx" podStartSLOduration=2.24295879 podStartE2EDuration="2.24295879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:05.241938601 +0000 UTC m=+1931.129844954" watchObservedRunningTime="2026-10-30 23:39:05.24295879 +0000 UTC m=+1931.130865143"
Oct 30 23:39:05.421410 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:05.421371    5078 patch_prober.go:29] interesting pod/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 23:39:05.421910 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:05.421434    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4" podUID="eb7f4356-b694-4ebb-a65e-8f7782591aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 23:39:05.923205 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:05.923171    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 23:39:05.923370 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:05.923222    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c" podUID="c8b4ee1d-90dd-416a-9955-cc54ad064723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 23:39:06.047776 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:06.047744    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d"]
Oct 30 23:39:06.051712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:06.051685    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a283f3ba-nfm8d"]
Oct 30 23:39:06.121791 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:06.121681    5078 generic.go:356] "Generic (PLEG): container finished" podID="472179d6-45c3-4598-8961-23c2f46b6fea" containerID="262afe6c9f0a38e124ba0fed3e5d8e9d711f4392bb9ba2064a5cbc7ed416f5d6" exitCode=0
Oct 30 23:39:06.121791 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:06.121719    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx" event={"ID":"472179d6-45c3-4598-8961-23c2f46b6fea","Type":"ContainerDied","Data":"262afe6c9f0a38e124ba0fed3e5d8e9d711f4392bb9ba2064a5cbc7ed416f5d6"}
Oct 30 23:39:06.319259 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:06.319230    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3afe0e20-7669-47b0-9ad4-6033155664a9" path="/var/lib/kubelet/pods/3afe0e20-7669-47b0-9ad4-6033155664a9/volumes"
Oct 30 23:39:06.420846 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:06.420807    5078 patch_prober.go:29] interesting pod/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 23:39:06.421047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:06.420859    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4" podUID="eb7f4356-b694-4ebb-a65e-8f7782591aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 23:39:06.922494 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:06.922458    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body=
Oct 30 23:39:06.922937 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:06.922512    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c" podUID="c8b4ee1d-90dd-416a-9955-cc54ad064723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused"
Oct 30 23:39:07.417140 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:07.417113    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx"
Oct 30 23:39:07.421442 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:07.421414    5078 patch_prober.go:29] interesting pod/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Oct 30 23:39:07.421613 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:07.421462    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4" podUID="eb7f4356-b694-4ebb-a65e-8f7782591aea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Oct 30 23:39:07.508341 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:07.508241    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/472179d6-45c3-4598-8961-23c2f46b6fea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472179d6-45c3-4598-8961-23c2f46b6fea-maas-api-xks-ca\") pod \"472179d6-45c3-4598-8961-23c2f46b6fea\" (UID: \"472179d6-45c3-4598-8961-23c2f46b6fea\") "
Oct 30 23:39:07.508526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:07.508501    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/472179d6-45c3-4598-8961-23c2f46b6fea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472179d6-45c3-4598-8961-23c2f46b6fea-maas-api-openshift-ca\") pod \"472179d6-45c3-4598-8961-23c2f46b6fea\" (UID: \"472179d6-45c3-4598-8961-23c2f46b6fea\") "
Oct 30 23:39:07.508784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:07.508763    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/472179d6-45c3-4598-8961-23c2f46b6fea-kube-api-access-d5647\" (UniqueName: \"kubernetes.io/projected/472179d6-45c3-4598-8961-23c2f46b6fea-kube-api-access-d5647\") pod \"472179d6-45c3-4598-8961-23c2f46b6fea\" (UID: \"472179d6-45c3-4598-8961-23c2f46b6fea\") "
Oct 30 23:39:07.509448 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:07.509417    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472179d6-45c3-4598-8961-23c2f46b6fea-maas-api-openshift-ca" pod "472179d6-45c3-4598-8961-23c2f46b6fea" (UID: "472179d6-45c3-4598-8961-23c2f46b6fea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:39:07.509976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:07.509947    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/472179d6-45c3-4598-8961-23c2f46b6fea-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:39:07.512323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:07.512235    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/472179d6-45c3-4598-8961-23c2f46b6fea-maas-api-xks-ca" pod "472179d6-45c3-4598-8961-23c2f46b6fea" (UID: "472179d6-45c3-4598-8961-23c2f46b6fea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:39:07.512897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:07.512856    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472179d6-45c3-4598-8961-23c2f46b6fea-kube-api-access-d5647" pod "472179d6-45c3-4598-8961-23c2f46b6fea" (UID: "472179d6-45c3-4598-8961-23c2f46b6fea"). InnerVolumeSpecName "kube-api-access-d5647". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:39:07.612283 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:07.612243    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5647\" (UniqueName: \"kubernetes.io/projected/472179d6-45c3-4598-8961-23c2f46b6fea-kube-api-access-d5647\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:39:07.612283 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:07.612287    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/472179d6-45c3-4598-8961-23c2f46b6fea-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:39:07.927104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:07.927071    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:39:07.928214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:07.928191    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:39:08.138935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:08.138912    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx"
Oct 30 23:39:08.138935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:08.138913    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx" event={"ID":"472179d6-45c3-4598-8961-23c2f46b6fea","Type":"ContainerDied","Data":"a353c5ee00e0cc47659a5079100a26b3f39348c784784a45f0ff5a14227a83d1"}
Oct 30 23:39:08.139155 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:08.138967    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a353c5ee00e0cc47659a5079100a26b3f39348c784784a45f0ff5a14227a83d1"
Oct 30 23:39:08.426646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:08.426616    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:39:08.428106 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:08.428083    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:39:16.046629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:16.046586    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2"]
Oct 30 23:39:16.096257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:16.096229    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37084415-f9px2"]
Oct 30 23:39:16.299120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:16.299030    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33d1f453-d599-4b5a-8701-76670832838e" path="/var/lib/kubelet/pods/33d1f453-d599-4b5a-8701-76670832838e/volumes"
Oct 30 23:39:17.591647 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:39:17.591580    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:39:38.314109 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.314075    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"]
Oct 30 23:39:38.314696 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.314680    5078 cpu_manager.go:414] "containerMap: removing container" podUID="472179d6-45c3-4598-8961-23c2f46b6fea" containerName="revoke-keys"
Oct 30 23:39:38.314696 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.314695    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472179d6-45c3-4598-8961-23c2f46b6fea" containerName="revoke-keys"
Oct 30 23:39:38.314820 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.314811    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="472179d6-45c3-4598-8961-23c2f46b6fea" containerName="revoke-keys"
Oct 30 23:39:38.318489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.318469    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.493632 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.489837    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7d1536-openshift-default-dockercfg-tl4jc\""
Oct 30 23:39:38.527426 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.495304    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-credential-socket\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.545755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.545711    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-podinfo\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.574673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.549904    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4539c4b3-4808-4706-b794-82d7a2fa6034-istiod-ca-cert\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.703618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.703271    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-workload-socket\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.703618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.703567    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-token\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.703891 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.703736    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-workload-certs\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.719647 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.704032    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnl24\" (UniqueName: \"kubernetes.io/projected/4539c4b3-4808-4706-b794-82d7a2fa6034-kube-api-access-rnl24\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.719647 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.704170    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-data\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.719647 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.704210    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-envoy\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.792897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.767099    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"]
Oct 30 23:39:38.818182 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.816965    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4539c4b3-4808-4706-b794-82d7a2fa6034-istiod-ca-cert\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.818182 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.817015    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-workload-socket\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.818182 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.817079    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-token\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.818182 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.817168    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-workload-certs\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.818182 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.817281    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnl24\" (UniqueName: \"kubernetes.io/projected/4539c4b3-4808-4706-b794-82d7a2fa6034-kube-api-access-rnl24\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.818182 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.817312    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-data\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.818182 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.817344    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-envoy\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.818182 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.817461    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-credential-socket\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.818182 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.817504    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-podinfo\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.819213 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.819181    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-workload-certs\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.820178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.820158    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4539c4b3-4808-4706-b794-82d7a2fa6034-istiod-ca-cert\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.821564 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.821538    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-workload-socket\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.822383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.822361    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-data\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.823655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.823442    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-credential-socket\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.868253 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.868171    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-podinfo\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.874402 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.874369    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-envoy\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.894185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.894124    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnl24\" (UniqueName: \"kubernetes.io/projected/4539c4b3-4808-4706-b794-82d7a2fa6034-kube-api-access-rnl24\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.896780 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.896732    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-token\") pod \"e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") " pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:38.980917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:38.980884    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:39.499867 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:39.499844    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"]
Oct 30 23:39:39.500556 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:39:39.500520    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4539c4b3_4808_4706_b794_82d7a2fa6034.slice/crio-f96669a80713b6f6c4e672603f958abd5ff03dbbd7e801f76bcaabfb9d653b64 WatchSource:0}: Error finding container f96669a80713b6f6c4e672603f958abd5ff03dbbd7e801f76bcaabfb9d653b64: Status 404 returned error can't find the container with id f96669a80713b6f6c4e672603f958abd5ff03dbbd7e801f76bcaabfb9d653b64
Oct 30 23:39:39.503492 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:39.503459    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:39:39.503616 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:39.503533    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:39:39.503616 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:39.503562    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:39:40.441796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:40.441754    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26" event={"ID":"4539c4b3-4808-4706-b794-82d7a2fa6034","Type":"ContainerStarted","Data":"370b58a9d4d9006ab5ca5ea920af855449df7322bc79d00cea53277ca522310c"}
Oct 30 23:39:40.441963 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:40.441818    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26" event={"ID":"4539c4b3-4808-4706-b794-82d7a2fa6034","Type":"ContainerStarted","Data":"f96669a80713b6f6c4e672603f958abd5ff03dbbd7e801f76bcaabfb9d653b64"}
Oct 30 23:39:40.611730 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:40.611683    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26" podStartSLOduration=2.6116684980000002 podStartE2EDuration="2.611668498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:40.607122487 +0000 UTC m=+1966.495028841" watchObservedRunningTime="2026-10-30 23:39:40.611668498 +0000 UTC m=+1966.499574850"
Oct 30 23:39:40.981801 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:40.981769    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:40.987901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:40.987877    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:41.450169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:41.450137    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:41.451095 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:41.451073    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:39:44.419909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.419840    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"]
Oct 30 23:39:44.424209 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.424178    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.444647 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.444590    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-credential-socket\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.444838 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.444820    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2a127b-3a86-45d1-8e4a-10557a96a711-istiod-ca-cert\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.444943 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.444930    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-workload-certs\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.444986 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.444975    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-data\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.445035 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.445023    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-workload-socket\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.445091 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.445080    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-token\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.445127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.445105    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-podinfo\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.445162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.445133    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5276\" (UniqueName: \"kubernetes.io/projected/ae2a127b-3a86-45d1-8e4a-10557a96a711-kube-api-access-w5276\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.445162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.445156    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-envoy\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.477047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.476972    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0a7530-openshift-default-dockercfg-znn9k\""
Oct 30 23:39:44.479898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.479870    5078 generic.go:356] "Generic (PLEG): container finished" podID="bca2f196-2daf-4071-b366-596e7ff0f23d" containerID="866067e7d180f1fefe39467b336b5e9a398c807897dd7dd24fd4c0fe6d92182a" exitCode=0
Oct 30 23:39:44.480088 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.480006    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg" event={"ID":"bca2f196-2daf-4071-b366-596e7ff0f23d","Type":"ContainerDied","Data":"866067e7d180f1fefe39467b336b5e9a398c807897dd7dd24fd4c0fe6d92182a"}
Oct 30 23:39:44.480545 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.480530    5078 scope.go:122] "RemoveContainer" containerID="866067e7d180f1fefe39467b336b5e9a398c807897dd7dd24fd4c0fe6d92182a"
Oct 30 23:39:44.482832 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.482807    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:39:44.482942 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.482891    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:39:44.483009 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.482939    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:39:44.532004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.531964    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"]
Oct 30 23:39:44.547636 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.547550    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-workload-certs\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.547945 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.547925    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-data\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.548797 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.548261    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-workload-certs\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.548945 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.548370    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-data\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.549242 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.549222    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-workload-socket\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.549484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.549465    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-token\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.549943 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.549924    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-podinfo\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.550164 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.549591    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-workload-socket\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.550267 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.550114    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5276\" (UniqueName: \"kubernetes.io/projected/ae2a127b-3a86-45d1-8e4a-10557a96a711-kube-api-access-w5276\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.550397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.550383    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-envoy\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.550530 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.550517    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-credential-socket\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.550900 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.550883    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2a127b-3a86-45d1-8e4a-10557a96a711-istiod-ca-cert\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.551092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.551068    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-credential-socket\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.552170 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.552149    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2a127b-3a86-45d1-8e4a-10557a96a711-istiod-ca-cert\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.553685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.553662    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-podinfo\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.554873 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.554851    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-envoy\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.716348 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.716282    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5276\" (UniqueName: \"kubernetes.io/projected/ae2a127b-3a86-45d1-8e4a-10557a96a711-kube-api-access-w5276\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:44.820683 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:44.817508    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-token\") pod \"e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") " pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:45.046621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:45.044573    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:45.491318 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:45.491282    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg" event={"ID":"bca2f196-2daf-4071-b366-596e7ff0f23d","Type":"ContainerStarted","Data":"b79fe0f55cf57f69637487be5c0dd6d37ed5762dc3ea8a751ec0276f48ef2df2"}
Oct 30 23:39:45.512699 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:45.512671    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"]
Oct 30 23:39:45.514347 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:39:45.514323    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2a127b_3a86_45d1_8e4a_10557a96a711.slice/crio-38e3ebcb36a2b7d4d1e4105aa86f456cf1e3fe550f8cd4b19dac44371e579340 WatchSource:0}: Error finding container 38e3ebcb36a2b7d4d1e4105aa86f456cf1e3fe550f8cd4b19dac44371e579340: Status 404 returned error can't find the container with id 38e3ebcb36a2b7d4d1e4105aa86f456cf1e3fe550f8cd4b19dac44371e579340
Oct 30 23:39:45.525688 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:45.525649    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:39:45.526022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:45.525995    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:39:45.526135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:45.526062    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:39:46.165622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:46.163916    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:39:46.181624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:46.179750    5078 patch_prober.go:29] interesting pod/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 23:39:46.181624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:46.179811    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg" podUID="bca2f196-2daf-4071-b366-596e7ff0f23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 23:39:46.499879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:46.499778    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6" event={"ID":"ae2a127b-3a86-45d1-8e4a-10557a96a711","Type":"ContainerStarted","Data":"aa59d354ec5d91a535b2c4bafd7d0a0c257a72098eb79864d3f9d74a7fda6edd"}
Oct 30 23:39:46.499879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:46.499837    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6" event={"ID":"ae2a127b-3a86-45d1-8e4a-10557a96a711","Type":"ContainerStarted","Data":"38e3ebcb36a2b7d4d1e4105aa86f456cf1e3fe550f8cd4b19dac44371e579340"}
Oct 30 23:39:46.620482 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:46.620422    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6" podStartSLOduration=2.62040408 podStartE2EDuration="2.62040408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:46.619496577 +0000 UTC m=+1972.507402932" watchObservedRunningTime="2026-10-30 23:39:46.62040408 +0000 UTC m=+1972.508310433"
Oct 30 23:39:47.045833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:47.045798    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:47.051809 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:47.051773    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:47.179621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:47.177703    5078 patch_prober.go:29] interesting pod/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 23:39:47.179621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:47.177760    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg" podUID="bca2f196-2daf-4071-b366-596e7ff0f23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 23:39:47.507376 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:47.507296    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:47.508274 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:47.508255    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:39:47.610930 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:39:47.610902    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:39:48.164672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:48.163481    5078 patch_prober.go:29] interesting pod/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 23:39:48.164672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:48.163563    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg" podUID="bca2f196-2daf-4071-b366-596e7ff0f23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 23:39:48.504676 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:48.502318    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt"]
Oct 30 23:39:48.697124 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:48.677813    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt"]
Oct 30 23:39:48.725727 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:48.684574    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt"
Oct 30 23:39:48.751481 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:48.751429    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl"]
Oct 30 23:39:48.823234 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:48.823201    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7d1536\""
Oct 30 23:39:48.833072 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:48.832928    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7d1536-dockercfg-67m27\""
Oct 30 23:39:48.853348 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:48.853316    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8395dcd7-d923-41f9-984e-ecf53b137eb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt\" (UID: \"8395dcd7-d923-41f9-984e-ecf53b137eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt"
Oct 30 23:39:48.858959 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:48.854053    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x527s\" (UniqueName: \"kubernetes.io/projected/8395dcd7-d923-41f9-984e-ecf53b137eb2-kube-api-access-x527s\") pod \"payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt\" (UID: \"8395dcd7-d923-41f9-984e-ecf53b137eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt"
Oct 30 23:39:48.962714 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:48.962675    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x527s\" (UniqueName: \"kubernetes.io/projected/8395dcd7-d923-41f9-984e-ecf53b137eb2-kube-api-access-x527s\") pod \"payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt\" (UID: \"8395dcd7-d923-41f9-984e-ecf53b137eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt"
Oct 30 23:39:48.962925 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:48.962726    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8395dcd7-d923-41f9-984e-ecf53b137eb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt\" (UID: \"8395dcd7-d923-41f9-984e-ecf53b137eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt"
Oct 30 23:39:48.963813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:48.963782    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8395dcd7-d923-41f9-984e-ecf53b137eb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt\" (UID: \"8395dcd7-d923-41f9-984e-ecf53b137eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt"
Oct 30 23:39:49.010655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.002119    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl"
Oct 30 23:39:49.074518 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.041700    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl"]
Oct 30 23:39:49.074518 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.042428    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:39:49.106463 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.106420    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:39:49.108028 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.106770    5078 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" probeResult="failure" output="Get \"https://10.133.0.84:8000/health\": context deadline exceeded"
Oct 30 23:39:49.136616 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.126869    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x527s\" (UniqueName: \"kubernetes.io/projected/8395dcd7-d923-41f9-984e-ecf53b137eb2-kube-api-access-x527s\") pod \"payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt\" (UID: \"8395dcd7-d923-41f9-984e-ecf53b137eb2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt"
Oct 30 23:39:49.138469 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.138395    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:39:49.162629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.161508    5078 patch_prober.go:29] interesting pod/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 23:39:49.162629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.161564    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg" podUID="bca2f196-2daf-4071-b366-596e7ff0f23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 23:39:49.164757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.164710    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt"
Oct 30 23:39:49.208834 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.208793    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260ce73f-2f49-43e7-a62f-9bcd51c63075-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7d1536-797545749-76cgl\" (UID: \"260ce73f-2f49-43e7-a62f-9bcd51c63075\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl"
Oct 30 23:39:49.209009 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.208939    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj7b8\" (UniqueName: \"kubernetes.io/projected/260ce73f-2f49-43e7-a62f-9bcd51c63075-kube-api-access-xj7b8\") pod \"payload-processing-e2e-inf-a-7d1536-797545749-76cgl\" (UID: \"260ce73f-2f49-43e7-a62f-9bcd51c63075\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl"
Oct 30 23:39:49.311251 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.310544    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260ce73f-2f49-43e7-a62f-9bcd51c63075-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7d1536-797545749-76cgl\" (UID: \"260ce73f-2f49-43e7-a62f-9bcd51c63075\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl"
Oct 30 23:39:49.311251 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.310633    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj7b8\" (UniqueName: \"kubernetes.io/projected/260ce73f-2f49-43e7-a62f-9bcd51c63075-kube-api-access-xj7b8\") pod \"payload-processing-e2e-inf-a-7d1536-797545749-76cgl\" (UID: \"260ce73f-2f49-43e7-a62f-9bcd51c63075\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl"
Oct 30 23:39:49.311533 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.311429    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260ce73f-2f49-43e7-a62f-9bcd51c63075-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7d1536-797545749-76cgl\" (UID: \"260ce73f-2f49-43e7-a62f-9bcd51c63075\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl"
Oct 30 23:39:49.384567 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.384489    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj7b8\" (UniqueName: \"kubernetes.io/projected/260ce73f-2f49-43e7-a62f-9bcd51c63075-kube-api-access-xj7b8\") pod \"payload-processing-e2e-inf-a-7d1536-797545749-76cgl\" (UID: \"260ce73f-2f49-43e7-a62f-9bcd51c63075\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl"
Oct 30 23:39:49.480623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.478153    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl"
Oct 30 23:39:49.611078 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.611043    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt"]
Oct 30 23:39:49.652743 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.652714    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt" event={"ID":"8395dcd7-d923-41f9-984e-ecf53b137eb2","Type":"ContainerStarted","Data":"ed43c1476438827b7ad6a6a7031c64e0646cf928654998818552cc303852f77e"}
Oct 30 23:39:49.887263 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:39:49.887232    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260ce73f_2f49_43e7_a62f_9bcd51c63075.slice/crio-322c4712ec6561cdaff6746aaeb785a120b19b1eb08fc121ed224cfd4fb94804 WatchSource:0}: Error finding container 322c4712ec6561cdaff6746aaeb785a120b19b1eb08fc121ed224cfd4fb94804: Status 404 returned error can't find the container with id 322c4712ec6561cdaff6746aaeb785a120b19b1eb08fc121ed224cfd4fb94804
Oct 30 23:39:49.887781 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:49.887493    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl"]
Oct 30 23:39:50.180678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:50.179969    5078 patch_prober.go:29] interesting pod/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 23:39:50.180678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:50.180030    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg" podUID="bca2f196-2daf-4071-b366-596e7ff0f23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 23:39:50.684619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:50.684119    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl" event={"ID":"260ce73f-2f49-43e7-a62f-9bcd51c63075","Type":"ContainerStarted","Data":"7dec1821bf7011f55a564b2b10d247755f777390014c9ccc7412b2ab5471ae89"}
Oct 30 23:39:50.684619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:50.684223    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl" event={"ID":"260ce73f-2f49-43e7-a62f-9bcd51c63075","Type":"ContainerStarted","Data":"322c4712ec6561cdaff6746aaeb785a120b19b1eb08fc121ed224cfd4fb94804"}
Oct 30 23:39:50.684619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:50.684336    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl"
Oct 30 23:39:50.686266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:50.686238    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt" event={"ID":"8395dcd7-d923-41f9-984e-ecf53b137eb2","Type":"ContainerStarted","Data":"137ccbbb3d53b718db77fe631f0844a33ebfbe02c959a78a29b1b82d214242bf"}
Oct 30 23:39:50.686391 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:50.686338    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt"
Oct 30 23:39:50.803121 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:50.803072    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl" podStartSLOduration=2.803055758 podStartE2EDuration="2.803055758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:50.799083729 +0000 UTC m=+1976.686990084" watchObservedRunningTime="2026-10-30 23:39:50.803055758 +0000 UTC m=+1976.690962111"
Oct 30 23:39:50.934201 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:50.933811    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt" podStartSLOduration=2.9337872799999998 podStartE2EDuration="2.93378728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:50.923414519 +0000 UTC m=+1976.811320873" watchObservedRunningTime="2026-10-30 23:39:50.93378728 +0000 UTC m=+1976.821693633"
Oct 30 23:39:51.160556 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:51.160507    5078 patch_prober.go:29] interesting pod/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 23:39:51.160889 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:51.160841    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg" podUID="bca2f196-2daf-4071-b366-596e7ff0f23d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 23:39:52.164176 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:52.164148    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:39:52.165280 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:52.165262    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:39:53.301064 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:53.301030    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp"]
Oct 30 23:39:53.305303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:53.305288    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp"
Oct 30 23:39:53.359041 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:53.359014    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0a7530\""
Oct 30 23:39:53.363042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:53.363014    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp"]
Oct 30 23:39:53.412886 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:53.412854    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0a7530-dockercfg-9xhtk\""
Oct 30 23:39:53.471683 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:53.468105    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f35153-159d-4249-86fe-fcab10ce168d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp\" (UID: \"63f35153-159d-4249-86fe-fcab10ce168d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp"
Oct 30 23:39:53.503622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:53.480157    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mnm9\" (UniqueName: \"kubernetes.io/projected/63f35153-159d-4249-86fe-fcab10ce168d-kube-api-access-9mnm9\") pod \"payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp\" (UID: \"63f35153-159d-4249-86fe-fcab10ce168d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp"
Oct 30 23:39:53.704550 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:53.704458    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mnm9\" (UniqueName: \"kubernetes.io/projected/63f35153-159d-4249-86fe-fcab10ce168d-kube-api-access-9mnm9\") pod \"payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp\" (UID: \"63f35153-159d-4249-86fe-fcab10ce168d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp"
Oct 30 23:39:53.711459 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:53.711423    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f35153-159d-4249-86fe-fcab10ce168d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp\" (UID: \"63f35153-159d-4249-86fe-fcab10ce168d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp"
Oct 30 23:39:53.775625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:53.774654    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f35153-159d-4249-86fe-fcab10ce168d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp\" (UID: \"63f35153-159d-4249-86fe-fcab10ce168d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp"
Oct 30 23:39:53.793129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:53.793089    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mnm9\" (UniqueName: \"kubernetes.io/projected/63f35153-159d-4249-86fe-fcab10ce168d-kube-api-access-9mnm9\") pod \"payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp\" (UID: \"63f35153-159d-4249-86fe-fcab10ce168d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp"
Oct 30 23:39:53.796650 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:53.796587    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s"]
Oct 30 23:39:53.801838 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:53.801817    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s"]
Oct 30 23:39:53.801943 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:53.801919    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s"
Oct 30 23:39:53.919542 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:53.919506    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp"
Oct 30 23:39:53.921103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:53.921076    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s\" (UID: \"dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9\") " pod="openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s"
Oct 30 23:39:53.921252 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:53.921232    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppb8z\" (UniqueName: \"kubernetes.io/projected/dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9-kube-api-access-ppb8z\") pod \"payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s\" (UID: \"dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9\") " pod="openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s"
Oct 30 23:39:54.023090 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:54.023057    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppb8z\" (UniqueName: \"kubernetes.io/projected/dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9-kube-api-access-ppb8z\") pod \"payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s\" (UID: \"dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9\") " pod="openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s"
Oct 30 23:39:54.023402 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:54.023383    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s\" (UID: \"dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9\") " pod="openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s"
Oct 30 23:39:54.024071 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:54.024047    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s\" (UID: \"dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9\") " pod="openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s"
Oct 30 23:39:54.080608 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:54.080560    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppb8z\" (UniqueName: \"kubernetes.io/projected/dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9-kube-api-access-ppb8z\") pod \"payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s\" (UID: \"dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9\") " pod="openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s"
Oct 30 23:39:54.149618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:54.148459    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s"
Oct 30 23:39:54.305528 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:54.305462    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp"]
Oct 30 23:39:54.308833 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:39:54.308810    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f35153_159d_4249_86fe_fcab10ce168d.slice/crio-16ab6a9ee04a7d194b5e0eb41b2061bbb8adfbfe3f5e4187f07c950372e8fe08 WatchSource:0}: Error finding container 16ab6a9ee04a7d194b5e0eb41b2061bbb8adfbfe3f5e4187f07c950372e8fe08: Status 404 returned error can't find the container with id 16ab6a9ee04a7d194b5e0eb41b2061bbb8adfbfe3f5e4187f07c950372e8fe08
Oct 30 23:39:54.569055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:54.569021    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s"]
Oct 30 23:39:54.572384 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:39:54.572362    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc94c3c7_8f6b_41aa_9d6f_42dbb71783e9.slice/crio-c73bbdd7f67b579abf043f927c2bb2928852a921364227d858292bd37f9b7536 WatchSource:0}: Error finding container c73bbdd7f67b579abf043f927c2bb2928852a921364227d858292bd37f9b7536: Status 404 returned error can't find the container with id c73bbdd7f67b579abf043f927c2bb2928852a921364227d858292bd37f9b7536
Oct 30 23:39:54.798773 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:54.798742    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp" event={"ID":"63f35153-159d-4249-86fe-fcab10ce168d","Type":"ContainerStarted","Data":"b7a258dce8fe640cd77c6884f745ad7c820e7492cd476065b452a63c11d6da47"}
Oct 30 23:39:54.798951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:54.798790    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp" event={"ID":"63f35153-159d-4249-86fe-fcab10ce168d","Type":"ContainerStarted","Data":"16ab6a9ee04a7d194b5e0eb41b2061bbb8adfbfe3f5e4187f07c950372e8fe08"}
Oct 30 23:39:54.799018 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:54.799000    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp"
Oct 30 23:39:54.800582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:54.800555    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s" event={"ID":"dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9","Type":"ContainerStarted","Data":"705daa61dd248037ce82a1c729786e1c6711057ee49128d473a1e938c066cb9c"}
Oct 30 23:39:54.800731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:54.800647    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s"
Oct 30 23:39:54.800731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:54.800673    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s" event={"ID":"dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9","Type":"ContainerStarted","Data":"c73bbdd7f67b579abf043f927c2bb2928852a921364227d858292bd37f9b7536"}
Oct 30 23:39:54.913453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:54.913362    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp" podStartSLOduration=1.913346555 podStartE2EDuration="1.913346555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:54.911275295 +0000 UTC m=+1980.799181647" watchObservedRunningTime="2026-10-30 23:39:54.913346555 +0000 UTC m=+1980.801252925"
Oct 30 23:39:55.028623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:39:55.027796    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s" podStartSLOduration=2.02777332 podStartE2EDuration="2.02777332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:39:55.026191604 +0000 UTC m=+1980.914097971" watchObservedRunningTime="2026-10-30 23:39:55.02777332 +0000 UTC m=+1980.915679666"
Oct 30 23:40:01.069472 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.063712    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4"]
Oct 30 23:40:01.072572 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.072543    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4"
Oct 30 23:40:01.124788 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.124750    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4"]
Oct 30 23:40:01.169480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.169442    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:40:01.169676 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.169499    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:40:01.169676 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.169537    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:40:01.169676 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.169548    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:40:01.173218 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.173195    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ea360a-a327-4d6c-aa7c-ac30541eba92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1ce657-847g4\" (UID: \"69ea360a-a327-4d6c-aa7c-ac30541eba92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4"
Oct 30 23:40:01.173343 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.173228    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ea360a-a327-4d6c-aa7c-ac30541eba92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1ce657-847g4\" (UID: \"69ea360a-a327-4d6c-aa7c-ac30541eba92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4"
Oct 30 23:40:01.173343 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.173253    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbfl5\" (UniqueName: \"kubernetes.io/projected/69ea360a-a327-4d6c-aa7c-ac30541eba92-kube-api-access-lbfl5\") pod \"maas-api-revoke-keys-e2e-api-a-1ce657-847g4\" (UID: \"69ea360a-a327-4d6c-aa7c-ac30541eba92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4"
Oct 30 23:40:01.274462 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.274431    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ea360a-a327-4d6c-aa7c-ac30541eba92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1ce657-847g4\" (UID: \"69ea360a-a327-4d6c-aa7c-ac30541eba92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4"
Oct 30 23:40:01.274462 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.274469    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ea360a-a327-4d6c-aa7c-ac30541eba92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1ce657-847g4\" (UID: \"69ea360a-a327-4d6c-aa7c-ac30541eba92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4"
Oct 30 23:40:01.274690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.274493    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbfl5\" (UniqueName: \"kubernetes.io/projected/69ea360a-a327-4d6c-aa7c-ac30541eba92-kube-api-access-lbfl5\") pod \"maas-api-revoke-keys-e2e-api-a-1ce657-847g4\" (UID: \"69ea360a-a327-4d6c-aa7c-ac30541eba92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4"
Oct 30 23:40:01.275159 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.275136    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ea360a-a327-4d6c-aa7c-ac30541eba92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1ce657-847g4\" (UID: \"69ea360a-a327-4d6c-aa7c-ac30541eba92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4"
Oct 30 23:40:01.276985 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.276963    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ea360a-a327-4d6c-aa7c-ac30541eba92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1ce657-847g4\" (UID: \"69ea360a-a327-4d6c-aa7c-ac30541eba92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4"
Oct 30 23:40:01.332469 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.332391    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbfl5\" (UniqueName: \"kubernetes.io/projected/69ea360a-a327-4d6c-aa7c-ac30541eba92-kube-api-access-lbfl5\") pod \"maas-api-revoke-keys-e2e-api-a-1ce657-847g4\" (UID: \"69ea360a-a327-4d6c-aa7c-ac30541eba92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4"
Oct 30 23:40:01.388075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.388028    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4"
Oct 30 23:40:01.695651 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.695538    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt"
Oct 30 23:40:01.695651 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.695631    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl"
Oct 30 23:40:01.798957 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.798891    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4"]
Oct 30 23:40:01.799105 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:40:01.799084    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ea360a_a327_4d6c_aa7c_ac30541eba92.slice/crio-13aab04b5dfd840612f86d476e82dc893d75aab92dbe005ec4a2a55391fff03d WatchSource:0}: Error finding container 13aab04b5dfd840612f86d476e82dc893d75aab92dbe005ec4a2a55391fff03d: Status 404 returned error can't find the container with id 13aab04b5dfd840612f86d476e82dc893d75aab92dbe005ec4a2a55391fff03d
Oct 30 23:40:01.852927 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:01.852896    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4" event={"ID":"69ea360a-a327-4d6c-aa7c-ac30541eba92","Type":"ContainerStarted","Data":"13aab04b5dfd840612f86d476e82dc893d75aab92dbe005ec4a2a55391fff03d"}
Oct 30 23:40:02.740541 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:02.740513    5078 scope.go:122] "RemoveContainer" containerID="68e56b01fdece1a29508789274a81a82d6e5473ff570b7a30ff1714b2b43f6da"
Oct 30 23:40:02.750970 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:02.750948    5078 scope.go:122] "RemoveContainer" containerID="99319e0f2c409812cda9889f6b3119dace4e6fe523d50f5d895944fb72a93334"
Oct 30 23:40:02.862863 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:02.862834    5078 generic.go:356] "Generic (PLEG): container finished" podID="69ea360a-a327-4d6c-aa7c-ac30541eba92" containerID="fa3563aef42b1a9936dbf6e5074a81ee8ae3ea932544b0e655c9823907e5be7f" exitCode=0
Oct 30 23:40:02.863021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:02.862914    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4" event={"ID":"69ea360a-a327-4d6c-aa7c-ac30541eba92","Type":"ContainerDied","Data":"fa3563aef42b1a9936dbf6e5074a81ee8ae3ea932544b0e655c9823907e5be7f"}
Oct 30 23:40:03.327987 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:03.327910    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"]
Oct 30 23:40:03.328156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:03.328089    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5" podUID="d861cce2-0e36-40ef-811f-e56b675031d8" containerName="istio-proxy" containerID="cri-o://3de9a68ebdfb1ee7599f7943a8042a8b2d4d99988360fe06f5020cc888a49a11" gracePeriod=30
Oct 30 23:40:04.075284 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:04.075254    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"]
Oct 30 23:40:04.076020 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:04.075989    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj" podUID="660a691a-733b-4ea7-a3db-4c55f649b099" containerName="istio-proxy" containerID="cri-o://b1346fa110f48ab56388373552c87d5c80fef4be79cac212bf5c6e86a09148a7" gracePeriod=30
Oct 30 23:40:04.211276 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:04.210759    5078 patch_prober.go:29] interesting pod/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:40:04.211276 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:04.210818    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5" podUID="d861cce2-0e36-40ef-811f-e56b675031d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:40:04.701632 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:04.701564    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4"
Oct 30 23:40:04.812626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:04.809268    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69ea360a-a327-4d6c-aa7c-ac30541eba92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ea360a-a327-4d6c-aa7c-ac30541eba92-maas-api-openshift-ca\") pod \"69ea360a-a327-4d6c-aa7c-ac30541eba92\" (UID: \"69ea360a-a327-4d6c-aa7c-ac30541eba92\") "
Oct 30 23:40:04.832535 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:04.832503    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69ea360a-a327-4d6c-aa7c-ac30541eba92-kube-api-access-lbfl5\" (UniqueName: \"kubernetes.io/projected/69ea360a-a327-4d6c-aa7c-ac30541eba92-kube-api-access-lbfl5\") pod \"69ea360a-a327-4d6c-aa7c-ac30541eba92\" (UID: \"69ea360a-a327-4d6c-aa7c-ac30541eba92\") "
Oct 30 23:40:04.833251 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:04.833221    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69ea360a-a327-4d6c-aa7c-ac30541eba92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ea360a-a327-4d6c-aa7c-ac30541eba92-maas-api-xks-ca\") pod \"69ea360a-a327-4d6c-aa7c-ac30541eba92\" (UID: \"69ea360a-a327-4d6c-aa7c-ac30541eba92\") "
Oct 30 23:40:04.834622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:04.832305    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ea360a-a327-4d6c-aa7c-ac30541eba92-maas-api-openshift-ca" pod "69ea360a-a327-4d6c-aa7c-ac30541eba92" (UID: "69ea360a-a327-4d6c-aa7c-ac30541eba92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:40:04.862212 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:04.862175    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69ea360a-a327-4d6c-aa7c-ac30541eba92-maas-api-xks-ca" pod "69ea360a-a327-4d6c-aa7c-ac30541eba92" (UID: "69ea360a-a327-4d6c-aa7c-ac30541eba92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:40:04.935842 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:04.923531    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ea360a-a327-4d6c-aa7c-ac30541eba92-kube-api-access-lbfl5" pod "69ea360a-a327-4d6c-aa7c-ac30541eba92" (UID: "69ea360a-a327-4d6c-aa7c-ac30541eba92"). InnerVolumeSpecName "kube-api-access-lbfl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:40:04.936036 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:04.935949    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69ea360a-a327-4d6c-aa7c-ac30541eba92-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:04.936036 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:04.935980    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69ea360a-a327-4d6c-aa7c-ac30541eba92-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:04.936036 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:04.935995    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbfl5\" (UniqueName: \"kubernetes.io/projected/69ea360a-a327-4d6c-aa7c-ac30541eba92-kube-api-access-lbfl5\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:04.954315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:04.954287    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4"
Oct 30 23:40:04.954478 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:04.954320    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4" event={"ID":"69ea360a-a327-4d6c-aa7c-ac30541eba92","Type":"ContainerDied","Data":"13aab04b5dfd840612f86d476e82dc893d75aab92dbe005ec4a2a55391fff03d"}
Oct 30 23:40:04.954478 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:04.954382    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13aab04b5dfd840612f86d476e82dc893d75aab92dbe005ec4a2a55391fff03d"
Oct 30 23:40:05.295469 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:05.295440    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"]
Oct 30 23:40:05.295912 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:05.295640    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs" podUID="1256cc85-9025-4545-949b-c37d21d8b052" containerName="istio-proxy" containerID="cri-o://f00b06c2aa8e5dd417e596760b95b3a6cac8e63a0e096fbe2250a07d868cf18b" gracePeriod=30
Oct 30 23:40:05.810734 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:05.810703    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s"
Oct 30 23:40:05.810909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:05.810784    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp"
Oct 30 23:40:06.632289 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:06.632240    5078 patch_prober.go:29] interesting pod/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:40:06.632753 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:06.632675    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj" podUID="660a691a-733b-4ea7-a3db-4c55f649b099" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:40:06.848045 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:06.848010    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:40:06.848218 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:06.848061    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs" podUID="1256cc85-9025-4545-949b-c37d21d8b052" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:40:07.335621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.331212    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"]
Oct 30 23:40:07.335621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.332435    5078 cpu_manager.go:414] "containerMap: removing container" podUID="69ea360a-a327-4d6c-aa7c-ac30541eba92" containerName="revoke-keys"
Oct 30 23:40:07.335621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.332458    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69ea360a-a327-4d6c-aa7c-ac30541eba92" containerName="revoke-keys"
Oct 30 23:40:07.335621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.332719    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="69ea360a-a327-4d6c-aa7c-ac30541eba92" containerName="revoke-keys"
Oct 30 23:40:07.621669 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.617753    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.774697 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.757852    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"]
Oct 30 23:40:07.774697 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.765996    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-26ce6b-openshift-default-dockercfg-tm9mm\""
Oct 30 23:40:07.786637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.786242    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-workload-certs\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.786637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.786366    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86r5d\" (UniqueName: \"kubernetes.io/projected/1ad7971e-1d0f-4404-883a-76bcf6da8b48-kube-api-access-86r5d\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.786637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.786474    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-credential-socket\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.786637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.786530    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-data\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.786637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.786631    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-token\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.787060 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.786861    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-workload-socket\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.787060 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.786894    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-envoy\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.787060 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.786927    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-podinfo\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.787060 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.786976    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istiod-ca-cert\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.891009 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.890931    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-workload-socket\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.891212 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.891195    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-envoy\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.891723 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.891330    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-podinfo\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.891723 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.891709    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istiod-ca-cert\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.891904 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.891772    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-workload-certs\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.891904 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.891867    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86r5d\" (UniqueName: \"kubernetes.io/projected/1ad7971e-1d0f-4404-883a-76bcf6da8b48-kube-api-access-86r5d\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.892013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.891973    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-credential-socket\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.892067 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.892021    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-data\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.892120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.892098    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-token\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.892944 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.892917    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-workload-socket\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.893889 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.893792    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-workload-certs\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.894501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.894473    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-credential-socket\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.894678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.894641    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istiod-ca-cert\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.894970 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.894953    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-data\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.895887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.895865    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-envoy\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.897921 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.897898    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-podinfo\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.951928 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.951894    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-token\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:07.952755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:07.952732    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86r5d\" (UniqueName: \"kubernetes.io/projected/1ad7971e-1d0f-4404-883a-76bcf6da8b48-kube-api-access-86r5d\") pod \"e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") " pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:08.088876 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.088844    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:08.544669 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.543649    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"]
Oct 30 23:40:08.593471 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:40:08.593422    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad7971e_1d0f_4404_883a_76bcf6da8b48.slice/crio-791c0e39bae0b9db94e346576426b232bfd4eea367f821f13b7925aa246a84ca WatchSource:0}: Error finding container 791c0e39bae0b9db94e346576426b232bfd4eea367f821f13b7925aa246a84ca: Status 404 returned error can't find the container with id 791c0e39bae0b9db94e346576426b232bfd4eea367f821f13b7925aa246a84ca
Oct 30 23:40:08.597665 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.597627    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:40:08.597770 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.597720    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:40:08.597770 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.597765    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:40:08.619312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.619289    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:40:08.705444 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.705372    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d861cce2-0e36-40ef-811f-e56b675031d8-kube-api-access-2lr9z\" (UniqueName: \"kubernetes.io/projected/d861cce2-0e36-40ef-811f-e56b675031d8-kube-api-access-2lr9z\") pod \"d861cce2-0e36-40ef-811f-e56b675031d8\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") "
Oct 30 23:40:08.705612 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.705462    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-credential-socket\") pod \"d861cce2-0e36-40ef-811f-e56b675031d8\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") "
Oct 30 23:40:08.705612 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.705570    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-workload-certs\") pod \"d861cce2-0e36-40ef-811f-e56b675031d8\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") "
Oct 30 23:40:08.705793 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.705773    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d861cce2-0e36-40ef-811f-e56b675031d8-istio-token\" (UniqueName: \"kubernetes.io/projected/d861cce2-0e36-40ef-811f-e56b675031d8-istio-token\") pod \"d861cce2-0e36-40ef-811f-e56b675031d8\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") "
Oct 30 23:40:08.705871 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.705805    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-credential-socket" pod "d861cce2-0e36-40ef-811f-e56b675031d8" (UID: "d861cce2-0e36-40ef-811f-e56b675031d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:08.705871 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.705822    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-workload-certs" pod "d861cce2-0e36-40ef-811f-e56b675031d8" (UID: "d861cce2-0e36-40ef-811f-e56b675031d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:08.705981 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.705873    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d861cce2-0e36-40ef-811f-e56b675031d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d861cce2-0e36-40ef-811f-e56b675031d8-istiod-ca-cert\") pod \"d861cce2-0e36-40ef-811f-e56b675031d8\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") "
Oct 30 23:40:08.705981 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.705899    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-istio-envoy\") pod \"d861cce2-0e36-40ef-811f-e56b675031d8\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") "
Oct 30 23:40:08.705981 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.705960    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-istio-data\") pod \"d861cce2-0e36-40ef-811f-e56b675031d8\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") "
Oct 30 23:40:08.706147 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.705994    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-workload-socket\") pod \"d861cce2-0e36-40ef-811f-e56b675031d8\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") "
Oct 30 23:40:08.706147 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.706034    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d861cce2-0e36-40ef-811f-e56b675031d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d861cce2-0e36-40ef-811f-e56b675031d8-istio-podinfo\") pod \"d861cce2-0e36-40ef-811f-e56b675031d8\" (UID: \"d861cce2-0e36-40ef-811f-e56b675031d8\") "
Oct 30 23:40:08.707967 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.706640    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-workload-socket" pod "d861cce2-0e36-40ef-811f-e56b675031d8" (UID: "d861cce2-0e36-40ef-811f-e56b675031d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:08.707967 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.706727    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-istio-data" pod "d861cce2-0e36-40ef-811f-e56b675031d8" (UID: "d861cce2-0e36-40ef-811f-e56b675031d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:08.707967 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.707405    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d861cce2-0e36-40ef-811f-e56b675031d8-istiod-ca-cert" pod "d861cce2-0e36-40ef-811f-e56b675031d8" (UID: "d861cce2-0e36-40ef-811f-e56b675031d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:40:08.707967 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.707508    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:08.707967 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.707542    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:08.707967 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.707558    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:08.707967 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.707573    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:08.708379 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.708269    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d861cce2-0e36-40ef-811f-e56b675031d8-kube-api-access-2lr9z" pod "d861cce2-0e36-40ef-811f-e56b675031d8" (UID: "d861cce2-0e36-40ef-811f-e56b675031d8"). InnerVolumeSpecName "kube-api-access-2lr9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:40:08.708379 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.708298    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d861cce2-0e36-40ef-811f-e56b675031d8-istio-token" pod "d861cce2-0e36-40ef-811f-e56b675031d8" (UID: "d861cce2-0e36-40ef-811f-e56b675031d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:40:08.709371 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.709347    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d861cce2-0e36-40ef-811f-e56b675031d8-istio-podinfo" pod "d861cce2-0e36-40ef-811f-e56b675031d8" (UID: "d861cce2-0e36-40ef-811f-e56b675031d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:40:08.709463 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.709451    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-istio-envoy" pod "d861cce2-0e36-40ef-811f-e56b675031d8" (UID: "d861cce2-0e36-40ef-811f-e56b675031d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:08.809523 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.809492    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d861cce2-0e36-40ef-811f-e56b675031d8-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:08.809523 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.809520    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d861cce2-0e36-40ef-811f-e56b675031d8-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:08.809523 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.809530    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d861cce2-0e36-40ef-811f-e56b675031d8-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:08.809989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.809539    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d861cce2-0e36-40ef-811f-e56b675031d8-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:08.809989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:08.809548    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lr9z\" (UniqueName: \"kubernetes.io/projected/d861cce2-0e36-40ef-811f-e56b675031d8-kube-api-access-2lr9z\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:09.030745 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.030658    5078 generic.go:356] "Generic (PLEG): container finished" podID="d861cce2-0e36-40ef-811f-e56b675031d8" containerID="3de9a68ebdfb1ee7599f7943a8042a8b2d4d99988360fe06f5020cc888a49a11" exitCode=0
Oct 30 23:40:09.030745 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.030727    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"
Oct 30 23:40:09.030962 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.030754    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5" event={"ID":"d861cce2-0e36-40ef-811f-e56b675031d8","Type":"ContainerDied","Data":"3de9a68ebdfb1ee7599f7943a8042a8b2d4d99988360fe06f5020cc888a49a11"}
Oct 30 23:40:09.030962 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.030812    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5" event={"ID":"d861cce2-0e36-40ef-811f-e56b675031d8","Type":"ContainerDied","Data":"2d47bd0f54851a1663017a29a49194e015839418b36ac769a0b86815f474acde"}
Oct 30 23:40:09.030962 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.030841    5078 scope.go:122] "RemoveContainer" containerID="3de9a68ebdfb1ee7599f7943a8042a8b2d4d99988360fe06f5020cc888a49a11"
Oct 30 23:40:09.032917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.032890    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9" event={"ID":"1ad7971e-1d0f-4404-883a-76bcf6da8b48","Type":"ContainerStarted","Data":"380ae4650e11749b1d329d95cffe581517f0a75900bfdb25ddb8feb200711d92"}
Oct 30 23:40:09.033122 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.033085    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9" event={"ID":"1ad7971e-1d0f-4404-883a-76bcf6da8b48","Type":"ContainerStarted","Data":"791c0e39bae0b9db94e346576426b232bfd4eea367f821f13b7925aa246a84ca"}
Oct 30 23:40:09.044097 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.044077    5078 scope.go:122] "RemoveContainer" containerID="3de9a68ebdfb1ee7599f7943a8042a8b2d4d99988360fe06f5020cc888a49a11"
Oct 30 23:40:09.044379 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:40:09.044352    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de9a68ebdfb1ee7599f7943a8042a8b2d4d99988360fe06f5020cc888a49a11\": container with ID starting with 3de9a68ebdfb1ee7599f7943a8042a8b2d4d99988360fe06f5020cc888a49a11 not found: ID does not exist" containerID="3de9a68ebdfb1ee7599f7943a8042a8b2d4d99988360fe06f5020cc888a49a11"
Oct 30 23:40:09.044461 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.044390    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de9a68ebdfb1ee7599f7943a8042a8b2d4d99988360fe06f5020cc888a49a11"} err="failed to get container status \"3de9a68ebdfb1ee7599f7943a8042a8b2d4d99988360fe06f5020cc888a49a11\": rpc error: code = NotFound desc = could not find container \"3de9a68ebdfb1ee7599f7943a8042a8b2d4d99988360fe06f5020cc888a49a11\": container with ID starting with 3de9a68ebdfb1ee7599f7943a8042a8b2d4d99988360fe06f5020cc888a49a11 not found: ID does not exist"
Oct 30 23:40:09.090629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.089727    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:09.091247 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.090970    5078 patch_prober.go:29] interesting pod/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 23:40:09.091247 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.091034    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9" podUID="1ad7971e-1d0f-4404-883a-76bcf6da8b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 23:40:09.206138 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.206075    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9" podStartSLOduration=2.20604893 podStartE2EDuration="2.20604893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:09.20438068 +0000 UTC m=+1995.092287035" watchObservedRunningTime="2026-10-30 23:40:09.20604893 +0000 UTC m=+1995.093955284"
Oct 30 23:40:09.353907 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.353869    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"]
Oct 30 23:40:09.404559 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.404528    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:40:09.429620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.417908    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-057303-openshift-default-5798ff8fd9-w6mn5"]
Oct 30 23:40:09.546328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.543049    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-istio-envoy\") pod \"660a691a-733b-4ea7-a3db-4c55f649b099\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") "
Oct 30 23:40:09.546328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.543110    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/660a691a-733b-4ea7-a3db-4c55f649b099-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660a691a-733b-4ea7-a3db-4c55f649b099-istio-podinfo\") pod \"660a691a-733b-4ea7-a3db-4c55f649b099\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") "
Oct 30 23:40:09.546328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.543141    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660a691a-733b-4ea7-a3db-4c55f649b099-istio-token\" (UniqueName: \"kubernetes.io/projected/660a691a-733b-4ea7-a3db-4c55f649b099-istio-token\") pod \"660a691a-733b-4ea7-a3db-4c55f649b099\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") "
Oct 30 23:40:09.546328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.543256    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660a691a-733b-4ea7-a3db-4c55f649b099-kube-api-access-2gtlx\" (UniqueName: \"kubernetes.io/projected/660a691a-733b-4ea7-a3db-4c55f649b099-kube-api-access-2gtlx\") pod \"660a691a-733b-4ea7-a3db-4c55f649b099\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") "
Oct 30 23:40:09.546328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.543352    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-istio-data\") pod \"660a691a-733b-4ea7-a3db-4c55f649b099\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") "
Oct 30 23:40:09.546328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.543393    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-workload-certs\") pod \"660a691a-733b-4ea7-a3db-4c55f649b099\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") "
Oct 30 23:40:09.546328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.543552    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-workload-socket\") pod \"660a691a-733b-4ea7-a3db-4c55f649b099\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") "
Oct 30 23:40:09.546328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.543613    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660a691a-733b-4ea7-a3db-4c55f649b099-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660a691a-733b-4ea7-a3db-4c55f649b099-istiod-ca-cert\") pod \"660a691a-733b-4ea7-a3db-4c55f649b099\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") "
Oct 30 23:40:09.546328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.543670    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-credential-socket\") pod \"660a691a-733b-4ea7-a3db-4c55f649b099\" (UID: \"660a691a-733b-4ea7-a3db-4c55f649b099\") "
Oct 30 23:40:09.546328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.544744    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-credential-socket" pod "660a691a-733b-4ea7-a3db-4c55f649b099" (UID: "660a691a-733b-4ea7-a3db-4c55f649b099"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:09.546328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.545314    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-istio-data" pod "660a691a-733b-4ea7-a3db-4c55f649b099" (UID: "660a691a-733b-4ea7-a3db-4c55f649b099"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:09.546328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.545433    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-workload-socket" pod "660a691a-733b-4ea7-a3db-4c55f649b099" (UID: "660a691a-733b-4ea7-a3db-4c55f649b099"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:09.546328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.545678    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-workload-certs" pod "660a691a-733b-4ea7-a3db-4c55f649b099" (UID: "660a691a-733b-4ea7-a3db-4c55f649b099"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:09.546328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.545976    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660a691a-733b-4ea7-a3db-4c55f649b099-istiod-ca-cert" pod "660a691a-733b-4ea7-a3db-4c55f649b099" (UID: "660a691a-733b-4ea7-a3db-4c55f649b099"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:40:09.549064 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.548941    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660a691a-733b-4ea7-a3db-4c55f649b099-kube-api-access-2gtlx" pod "660a691a-733b-4ea7-a3db-4c55f649b099" (UID: "660a691a-733b-4ea7-a3db-4c55f649b099"). InnerVolumeSpecName "kube-api-access-2gtlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:40:09.551771 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.549409    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/660a691a-733b-4ea7-a3db-4c55f649b099-istio-podinfo" pod "660a691a-733b-4ea7-a3db-4c55f649b099" (UID: "660a691a-733b-4ea7-a3db-4c55f649b099"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:40:09.551771 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.549504    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-istio-envoy" pod "660a691a-733b-4ea7-a3db-4c55f649b099" (UID: "660a691a-733b-4ea7-a3db-4c55f649b099"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:09.551771 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.549557    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660a691a-733b-4ea7-a3db-4c55f649b099-istio-token" pod "660a691a-733b-4ea7-a3db-4c55f649b099" (UID: "660a691a-733b-4ea7-a3db-4c55f649b099"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:40:09.650626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.650557    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gtlx\" (UniqueName: \"kubernetes.io/projected/660a691a-733b-4ea7-a3db-4c55f649b099-kube-api-access-2gtlx\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:09.650969 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.650942    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:09.650969 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.650972    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:09.651126 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.650988    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:09.651126 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.651001    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660a691a-733b-4ea7-a3db-4c55f649b099-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:09.651126 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.651015    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:09.651126 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.651029    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660a691a-733b-4ea7-a3db-4c55f649b099-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:09.651126 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.651042    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660a691a-733b-4ea7-a3db-4c55f649b099-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:09.651126 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:09.651056    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660a691a-733b-4ea7-a3db-4c55f649b099-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:10.040952 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.040924    5078 generic.go:356] "Generic (PLEG): container finished" podID="660a691a-733b-4ea7-a3db-4c55f649b099" containerID="b1346fa110f48ab56388373552c87d5c80fef4be79cac212bf5c6e86a09148a7" exitCode=0
Oct 30 23:40:10.041419 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.040983    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"
Oct 30 23:40:10.041419 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.041006    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj" event={"ID":"660a691a-733b-4ea7-a3db-4c55f649b099","Type":"ContainerDied","Data":"b1346fa110f48ab56388373552c87d5c80fef4be79cac212bf5c6e86a09148a7"}
Oct 30 23:40:10.041419 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.041083    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj" event={"ID":"660a691a-733b-4ea7-a3db-4c55f649b099","Type":"ContainerDied","Data":"90def38b0b85eb0709f493408fb11e7bc33644d3438694ce1b181d6c0176808e"}
Oct 30 23:40:10.041419 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.041106    5078 scope.go:122] "RemoveContainer" containerID="b1346fa110f48ab56388373552c87d5c80fef4be79cac212bf5c6e86a09148a7"
Oct 30 23:40:10.056295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.056273    5078 scope.go:122] "RemoveContainer" containerID="b1346fa110f48ab56388373552c87d5c80fef4be79cac212bf5c6e86a09148a7"
Oct 30 23:40:10.070679 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:40:10.056619    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1346fa110f48ab56388373552c87d5c80fef4be79cac212bf5c6e86a09148a7\": container with ID starting with b1346fa110f48ab56388373552c87d5c80fef4be79cac212bf5c6e86a09148a7 not found: ID does not exist" containerID="b1346fa110f48ab56388373552c87d5c80fef4be79cac212bf5c6e86a09148a7"
Oct 30 23:40:10.070679 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.056659    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1346fa110f48ab56388373552c87d5c80fef4be79cac212bf5c6e86a09148a7"} err="failed to get container status \"b1346fa110f48ab56388373552c87d5c80fef4be79cac212bf5c6e86a09148a7\": rpc error: code = NotFound desc = could not find container \"b1346fa110f48ab56388373552c87d5c80fef4be79cac212bf5c6e86a09148a7\": container with ID starting with b1346fa110f48ab56388373552c87d5c80fef4be79cac212bf5c6e86a09148a7 not found: ID does not exist"
Oct 30 23:40:10.183680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.170113    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:10.216463 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.216424    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"]
Oct 30 23:40:10.220206 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.220177    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-tqrvj"]
Oct 30 23:40:10.309031 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.306080    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660a691a-733b-4ea7-a3db-4c55f649b099" path="/var/lib/kubelet/pods/660a691a-733b-4ea7-a3db-4c55f649b099/volumes"
Oct 30 23:40:10.309031 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.306921    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d861cce2-0e36-40ef-811f-e56b675031d8" path="/var/lib/kubelet/pods/d861cce2-0e36-40ef-811f-e56b675031d8/volumes"
Oct 30 23:40:10.624456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.624431    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:40:10.794309 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.794273    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-istio-data\") pod \"1256cc85-9025-4545-949b-c37d21d8b052\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") "
Oct 30 23:40:10.794506 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.794427    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1256cc85-9025-4545-949b-c37d21d8b052-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1256cc85-9025-4545-949b-c37d21d8b052-istio-podinfo\") pod \"1256cc85-9025-4545-949b-c37d21d8b052\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") "
Oct 30 23:40:10.794506 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.794476    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-workload-socket\") pod \"1256cc85-9025-4545-949b-c37d21d8b052\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") "
Oct 30 23:40:10.794652 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.794509    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1256cc85-9025-4545-949b-c37d21d8b052-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1256cc85-9025-4545-949b-c37d21d8b052-istiod-ca-cert\") pod \"1256cc85-9025-4545-949b-c37d21d8b052\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") "
Oct 30 23:40:10.794652 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.794556    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-istio-data" pod "1256cc85-9025-4545-949b-c37d21d8b052" (UID: "1256cc85-9025-4545-949b-c37d21d8b052"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:10.794652 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.794616    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-credential-socket\") pod \"1256cc85-9025-4545-949b-c37d21d8b052\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") "
Oct 30 23:40:10.794896 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.794665    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-istio-envoy\") pod \"1256cc85-9025-4545-949b-c37d21d8b052\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") "
Oct 30 23:40:10.794896 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.794698    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1256cc85-9025-4545-949b-c37d21d8b052-kube-api-access-2pp77\" (UniqueName: \"kubernetes.io/projected/1256cc85-9025-4545-949b-c37d21d8b052-kube-api-access-2pp77\") pod \"1256cc85-9025-4545-949b-c37d21d8b052\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") "
Oct 30 23:40:10.794896 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.794757    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-workload-socket" pod "1256cc85-9025-4545-949b-c37d21d8b052" (UID: "1256cc85-9025-4545-949b-c37d21d8b052"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:10.794896 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.794834    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1256cc85-9025-4545-949b-c37d21d8b052-istio-token\" (UniqueName: \"kubernetes.io/projected/1256cc85-9025-4545-949b-c37d21d8b052-istio-token\") pod \"1256cc85-9025-4545-949b-c37d21d8b052\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") "
Oct 30 23:40:10.795109 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.794924    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-workload-certs\") pod \"1256cc85-9025-4545-949b-c37d21d8b052\" (UID: \"1256cc85-9025-4545-949b-c37d21d8b052\") "
Oct 30 23:40:10.795109 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.794927    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-credential-socket" pod "1256cc85-9025-4545-949b-c37d21d8b052" (UID: "1256cc85-9025-4545-949b-c37d21d8b052"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:10.795214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.795177    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-workload-certs" pod "1256cc85-9025-4545-949b-c37d21d8b052" (UID: "1256cc85-9025-4545-949b-c37d21d8b052"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:10.795274 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.795215    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1256cc85-9025-4545-949b-c37d21d8b052-istiod-ca-cert" pod "1256cc85-9025-4545-949b-c37d21d8b052" (UID: "1256cc85-9025-4545-949b-c37d21d8b052"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:40:10.795921 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.795891    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:10.796047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.795924    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:10.796047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.795939    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:10.796047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.795953    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:10.796047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.795969    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1256cc85-9025-4545-949b-c37d21d8b052-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:10.797470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.797438    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-istio-envoy" pod "1256cc85-9025-4545-949b-c37d21d8b052" (UID: "1256cc85-9025-4545-949b-c37d21d8b052"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:10.797809 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.797788    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1256cc85-9025-4545-949b-c37d21d8b052-istio-podinfo" pod "1256cc85-9025-4545-949b-c37d21d8b052" (UID: "1256cc85-9025-4545-949b-c37d21d8b052"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:40:10.797974 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.797956    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1256cc85-9025-4545-949b-c37d21d8b052-istio-token" pod "1256cc85-9025-4545-949b-c37d21d8b052" (UID: "1256cc85-9025-4545-949b-c37d21d8b052"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:40:10.798281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.798262    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1256cc85-9025-4545-949b-c37d21d8b052-kube-api-access-2pp77" pod "1256cc85-9025-4545-949b-c37d21d8b052" (UID: "1256cc85-9025-4545-949b-c37d21d8b052"). InnerVolumeSpecName "kube-api-access-2pp77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:40:10.897822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.897744    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1256cc85-9025-4545-949b-c37d21d8b052-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:10.897822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.897773    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1256cc85-9025-4545-949b-c37d21d8b052-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:10.897822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.897783    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1256cc85-9025-4545-949b-c37d21d8b052-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:10.897822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:10.897792    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pp77\" (UniqueName: \"kubernetes.io/projected/1256cc85-9025-4545-949b-c37d21d8b052-kube-api-access-2pp77\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:11.051414 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:11.051380    5078 generic.go:356] "Generic (PLEG): container finished" podID="1256cc85-9025-4545-949b-c37d21d8b052" containerID="f00b06c2aa8e5dd417e596760b95b3a6cac8e63a0e096fbe2250a07d868cf18b" exitCode=0
Oct 30 23:40:11.051931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:11.051436    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"
Oct 30 23:40:11.051931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:11.051464    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs" event={"ID":"1256cc85-9025-4545-949b-c37d21d8b052","Type":"ContainerDied","Data":"f00b06c2aa8e5dd417e596760b95b3a6cac8e63a0e096fbe2250a07d868cf18b"}
Oct 30 23:40:11.051931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:11.051518    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs" event={"ID":"1256cc85-9025-4545-949b-c37d21d8b052","Type":"ContainerDied","Data":"90f13a5d2037a0cc234c58ac2675336825397b1629c448e48f1b6cad26914f74"}
Oct 30 23:40:11.051931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:11.051547    5078 scope.go:122] "RemoveContainer" containerID="f00b06c2aa8e5dd417e596760b95b3a6cac8e63a0e096fbe2250a07d868cf18b"
Oct 30 23:40:11.052946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:11.052926    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:11.053862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:11.053832    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:40:11.090232 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:11.090210    5078 scope.go:122] "RemoveContainer" containerID="f00b06c2aa8e5dd417e596760b95b3a6cac8e63a0e096fbe2250a07d868cf18b"
Oct 30 23:40:11.090554 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:40:11.090533    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f00b06c2aa8e5dd417e596760b95b3a6cac8e63a0e096fbe2250a07d868cf18b\": container with ID starting with f00b06c2aa8e5dd417e596760b95b3a6cac8e63a0e096fbe2250a07d868cf18b not found: ID does not exist" containerID="f00b06c2aa8e5dd417e596760b95b3a6cac8e63a0e096fbe2250a07d868cf18b"
Oct 30 23:40:11.090658 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:11.090562    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f00b06c2aa8e5dd417e596760b95b3a6cac8e63a0e096fbe2250a07d868cf18b"} err="failed to get container status \"f00b06c2aa8e5dd417e596760b95b3a6cac8e63a0e096fbe2250a07d868cf18b\": rpc error: code = NotFound desc = could not find container \"f00b06c2aa8e5dd417e596760b95b3a6cac8e63a0e096fbe2250a07d868cf18b\": container with ID starting with f00b06c2aa8e5dd417e596760b95b3a6cac8e63a0e096fbe2250a07d868cf18b not found: ID does not exist"
Oct 30 23:40:11.217287 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:11.217119    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"]
Oct 30 23:40:11.280586 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:11.280553    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-hsfcs"]
Oct 30 23:40:12.342622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:12.329589    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1256cc85-9025-4545-949b-c37d21d8b052" path="/var/lib/kubelet/pods/1256cc85-9025-4545-949b-c37d21d8b052/volumes"
Oct 30 23:40:16.092291 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:16.092258    5078 generic.go:356] "Generic (PLEG): container finished" podID="4cf62568-28ad-43c6-a0ab-f1680ef1a72c" containerID="dadf087ab6b027c3f6a6f2cc1e9e2daf54de156826205181c8f938b27e1afc16" exitCode=0
Oct 30 23:40:16.092776 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:16.092337    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv" event={"ID":"4cf62568-28ad-43c6-a0ab-f1680ef1a72c","Type":"ContainerDied","Data":"dadf087ab6b027c3f6a6f2cc1e9e2daf54de156826205181c8f938b27e1afc16"}
Oct 30 23:40:16.092823 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:16.092782    5078 scope.go:122] "RemoveContainer" containerID="dadf087ab6b027c3f6a6f2cc1e9e2daf54de156826205181c8f938b27e1afc16"
Oct 30 23:40:16.094440 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:16.094410    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:40:16.094506 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:16.094482    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:40:16.094551 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:16.094517    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:40:17.101795 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:17.101763    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv" event={"ID":"4cf62568-28ad-43c6-a0ab-f1680ef1a72c","Type":"ContainerStarted","Data":"7f56c1761fcc608527aed38a6bc595c602d514bce08473de1db7effb2cd8c077"}
Oct 30 23:40:17.612847 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:40:17.612810    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:40:17.927530 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:17.927448    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:40:17.927757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:17.927631    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 23:40:17.927757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:17.927693    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv" podUID="4cf62568-28ad-43c6-a0ab-f1680ef1a72c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 23:40:18.919659 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:18.919617    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"]
Oct 30 23:40:18.920105 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:18.919856    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r" podUID="d730be04-38d0-48a6-8235-20372487491f" containerName="istio-proxy" containerID="cri-o://dfe2a6b1451f0bec2f73294a0ef22e7253ef95e5f5aa083013f4f8478d778b38" gracePeriod=30
Oct 30 23:40:18.927673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:18.927649    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 23:40:18.927777 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:18.927696    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv" podUID="4cf62568-28ad-43c6-a0ab-f1680ef1a72c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 23:40:19.494091 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:19.494064    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"]
Oct 30 23:40:19.494274 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:19.494254    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch" podUID="c26c4072-f01f-45ae-bb38-4e62b8e87617" containerName="istio-proxy" containerID="cri-o://520e91a0e5734fe74c1ec55bb42716604d0e61a0b71d9b7d28edf926e08b7087" gracePeriod=30
Oct 30 23:40:19.927615 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:19.927555    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 23:40:19.928106 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:19.927634    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv" podUID="4cf62568-28ad-43c6-a0ab-f1680ef1a72c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 23:40:20.247268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:20.247173    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"]
Oct 30 23:40:20.339534 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:20.339491    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg" podUID="beb7cc10-970b-4a76-9f24-9151f8c7c0c8" containerName="istio-proxy" containerID="cri-o://020f0bbad6d3610327973300fc5470188ec173d9f55015e27dfe19e214ec4ef1" gracePeriod=30
Oct 30 23:40:20.752506 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:20.752471    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:40:20.752715 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:20.752547    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch" podUID="c26c4072-f01f-45ae-bb38-4e62b8e87617" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:40:20.957619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:20.948369    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 23:40:20.957619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:20.948444    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv" podUID="4cf62568-28ad-43c6-a0ab-f1680ef1a72c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 23:40:20.992701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:20.992661    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j"]
Oct 30 23:40:20.993930 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:20.993905    5078 cpu_manager.go:414] "containerMap: removing container" podUID="660a691a-733b-4ea7-a3db-4c55f649b099" containerName="istio-proxy"
Oct 30 23:40:20.994095 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:20.994082    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660a691a-733b-4ea7-a3db-4c55f649b099" containerName="istio-proxy"
Oct 30 23:40:20.994206 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:20.994196    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1256cc85-9025-4545-949b-c37d21d8b052" containerName="istio-proxy"
Oct 30 23:40:20.994286 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:20.994276    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1256cc85-9025-4545-949b-c37d21d8b052" containerName="istio-proxy"
Oct 30 23:40:20.994432 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:20.994421    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d861cce2-0e36-40ef-811f-e56b675031d8" containerName="istio-proxy"
Oct 30 23:40:20.994520 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:20.994510    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d861cce2-0e36-40ef-811f-e56b675031d8" containerName="istio-proxy"
Oct 30 23:40:20.994807 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:20.994792    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d861cce2-0e36-40ef-811f-e56b675031d8" containerName="istio-proxy"
Oct 30 23:40:20.994924 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:20.994914    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="660a691a-733b-4ea7-a3db-4c55f649b099" containerName="istio-proxy"
Oct 30 23:40:20.995013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:20.995004    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1256cc85-9025-4545-949b-c37d21d8b052" containerName="istio-proxy"
Oct 30 23:40:21.060680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:21.046919    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j"]
Oct 30 23:40:21.060680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:21.047321    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j"
Oct 30 23:40:21.098621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:21.094634    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33314a36-830a-4395-8f74-89227b549913-plugins-config-volume\") pod \"payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j\" (UID: \"33314a36-830a-4395-8f74-89227b549913\") " pod="openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j"
Oct 30 23:40:21.098621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:21.095011    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kht5\" (UniqueName: \"kubernetes.io/projected/33314a36-830a-4395-8f74-89227b549913-kube-api-access-4kht5\") pod \"payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j\" (UID: \"33314a36-830a-4395-8f74-89227b549913\") " pod="openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j"
Oct 30 23:40:21.115443 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:21.115411    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-26ce6b-dockercfg-swjlf\""
Oct 30 23:40:21.118433 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:21.115807    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-26ce6b\""
Oct 30 23:40:21.196937 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:21.196908    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kht5\" (UniqueName: \"kubernetes.io/projected/33314a36-830a-4395-8f74-89227b549913-kube-api-access-4kht5\") pod \"payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j\" (UID: \"33314a36-830a-4395-8f74-89227b549913\") " pod="openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j"
Oct 30 23:40:21.197179 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:21.197164    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33314a36-830a-4395-8f74-89227b549913-plugins-config-volume\") pod \"payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j\" (UID: \"33314a36-830a-4395-8f74-89227b549913\") " pod="openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j"
Oct 30 23:40:21.198072 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:21.198052    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33314a36-830a-4395-8f74-89227b549913-plugins-config-volume\") pod \"payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j\" (UID: \"33314a36-830a-4395-8f74-89227b549913\") " pod="openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j"
Oct 30 23:40:21.254378 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:21.254358    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kht5\" (UniqueName: \"kubernetes.io/projected/33314a36-830a-4395-8f74-89227b549913-kube-api-access-4kht5\") pod \"payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j\" (UID: \"33314a36-830a-4395-8f74-89227b549913\") " pod="openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j"
Oct 30 23:40:21.421869 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:21.421783    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j"
Oct 30 23:40:21.826911 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:21.826881    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j"]
Oct 30 23:40:21.827342 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:40:21.827322    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33314a36_830a_4395_8f74_89227b549913.slice/crio-0729890d198745fd220d8f9c17a458d93e687e84b03f84606b7dbdd2c20f77e4 WatchSource:0}: Error finding container 0729890d198745fd220d8f9c17a458d93e687e84b03f84606b7dbdd2c20f77e4: Status 404 returned error can't find the container with id 0729890d198745fd220d8f9c17a458d93e687e84b03f84606b7dbdd2c20f77e4
Oct 30 23:40:21.927705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:21.927678    5078 patch_prober.go:29] interesting pod/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 23:40:21.927824 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:21.927729    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv" podUID="4cf62568-28ad-43c6-a0ab-f1680ef1a72c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 23:40:21.953996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:21.953971    5078 patch_prober.go:29] interesting pod/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:40:21.954103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:21.954018    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg" podUID="beb7cc10-970b-4a76-9f24-9151f8c7c0c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:40:22.188851 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:22.188754    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j" event={"ID":"33314a36-830a-4395-8f74-89227b549913","Type":"ContainerStarted","Data":"ab7978292402368340713e4f0d991d4cabee65741fd57e5a0388d70abf3b246d"}
Oct 30 23:40:22.188851 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:22.188817    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j" event={"ID":"33314a36-830a-4395-8f74-89227b549913","Type":"ContainerStarted","Data":"0729890d198745fd220d8f9c17a458d93e687e84b03f84606b7dbdd2c20f77e4"}
Oct 30 23:40:22.189376 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:22.188994    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j"
Oct 30 23:40:22.295524 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:22.295185    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:40:22.295524 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:22.295256    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r" podUID="d730be04-38d0-48a6-8235-20372487491f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:40:22.324737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:22.319788    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j" podStartSLOduration=2.319763799 podStartE2EDuration="2.319763799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:22.311986229 +0000 UTC m=+2008.199892583" watchObservedRunningTime="2026-10-30 23:40:22.319763799 +0000 UTC m=+2008.207670155"
Oct 30 23:40:23.018662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:23.018346    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:40:23.018939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:23.018911    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:40:23.020435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:23.020410    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:40:24.189950 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.189878    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:40:24.205075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.205042    5078 generic.go:356] "Generic (PLEG): container finished" podID="d730be04-38d0-48a6-8235-20372487491f" containerID="dfe2a6b1451f0bec2f73294a0ef22e7253ef95e5f5aa083013f4f8478d778b38" exitCode=0
Oct 30 23:40:24.205237 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.205088    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"
Oct 30 23:40:24.205237 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.205120    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r" event={"ID":"d730be04-38d0-48a6-8235-20372487491f","Type":"ContainerDied","Data":"dfe2a6b1451f0bec2f73294a0ef22e7253ef95e5f5aa083013f4f8478d778b38"}
Oct 30 23:40:24.205237 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.205174    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r" event={"ID":"d730be04-38d0-48a6-8235-20372487491f","Type":"ContainerDied","Data":"21ffe91414717bf686c82aff297089d139ba65d25f1bfcf01d2e68bb9338f7fb"}
Oct 30 23:40:24.205237 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.205197    5078 scope.go:122] "RemoveContainer" containerID="dfe2a6b1451f0bec2f73294a0ef22e7253ef95e5f5aa083013f4f8478d778b38"
Oct 30 23:40:24.215165 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.215146    5078 scope.go:122] "RemoveContainer" containerID="dfe2a6b1451f0bec2f73294a0ef22e7253ef95e5f5aa083013f4f8478d778b38"
Oct 30 23:40:24.215461 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:40:24.215441    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe2a6b1451f0bec2f73294a0ef22e7253ef95e5f5aa083013f4f8478d778b38\": container with ID starting with dfe2a6b1451f0bec2f73294a0ef22e7253ef95e5f5aa083013f4f8478d778b38 not found: ID does not exist" containerID="dfe2a6b1451f0bec2f73294a0ef22e7253ef95e5f5aa083013f4f8478d778b38"
Oct 30 23:40:24.215571 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.215466    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe2a6b1451f0bec2f73294a0ef22e7253ef95e5f5aa083013f4f8478d778b38"} err="failed to get container status \"dfe2a6b1451f0bec2f73294a0ef22e7253ef95e5f5aa083013f4f8478d778b38\": rpc error: code = NotFound desc = could not find container \"dfe2a6b1451f0bec2f73294a0ef22e7253ef95e5f5aa083013f4f8478d778b38\": container with ID starting with dfe2a6b1451f0bec2f73294a0ef22e7253ef95e5f5aa083013f4f8478d778b38 not found: ID does not exist"
Oct 30 23:40:24.264575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.264551    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-credential-socket\") pod \"d730be04-38d0-48a6-8235-20372487491f\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") "
Oct 30 23:40:24.264709 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.264639    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d730be04-38d0-48a6-8235-20372487491f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d730be04-38d0-48a6-8235-20372487491f-istiod-ca-cert\") pod \"d730be04-38d0-48a6-8235-20372487491f\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") "
Oct 30 23:40:24.264709 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.264692    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d730be04-38d0-48a6-8235-20372487491f-istio-token\" (UniqueName: \"kubernetes.io/projected/d730be04-38d0-48a6-8235-20372487491f-istio-token\") pod \"d730be04-38d0-48a6-8235-20372487491f\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") "
Oct 30 23:40:24.264824 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.264715    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-istio-envoy\") pod \"d730be04-38d0-48a6-8235-20372487491f\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") "
Oct 30 23:40:24.264878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.264818    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-credential-socket" pod "d730be04-38d0-48a6-8235-20372487491f" (UID: "d730be04-38d0-48a6-8235-20372487491f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:24.264958 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.264939    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-workload-certs\") pod \"d730be04-38d0-48a6-8235-20372487491f\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") "
Oct 30 23:40:24.265019 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.265005    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d730be04-38d0-48a6-8235-20372487491f-istiod-ca-cert" pod "d730be04-38d0-48a6-8235-20372487491f" (UID: "d730be04-38d0-48a6-8235-20372487491f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:40:24.265093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.265060    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-workload-socket\") pod \"d730be04-38d0-48a6-8235-20372487491f\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") "
Oct 30 23:40:24.265273 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.265155    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-istio-data\") pod \"d730be04-38d0-48a6-8235-20372487491f\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") "
Oct 30 23:40:24.265273 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.265189    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d730be04-38d0-48a6-8235-20372487491f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d730be04-38d0-48a6-8235-20372487491f-istio-podinfo\") pod \"d730be04-38d0-48a6-8235-20372487491f\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") "
Oct 30 23:40:24.265273 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.265192    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-workload-certs" pod "d730be04-38d0-48a6-8235-20372487491f" (UID: "d730be04-38d0-48a6-8235-20372487491f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:24.265273 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.265251    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d730be04-38d0-48a6-8235-20372487491f-kube-api-access-rgg5q\" (UniqueName: \"kubernetes.io/projected/d730be04-38d0-48a6-8235-20372487491f-kube-api-access-rgg5q\") pod \"d730be04-38d0-48a6-8235-20372487491f\" (UID: \"d730be04-38d0-48a6-8235-20372487491f\") "
Oct 30 23:40:24.265691 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.265306    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-workload-socket" pod "d730be04-38d0-48a6-8235-20372487491f" (UID: "d730be04-38d0-48a6-8235-20372487491f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:24.265691 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.265386    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-istio-data" pod "d730be04-38d0-48a6-8235-20372487491f" (UID: "d730be04-38d0-48a6-8235-20372487491f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:24.266416 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.266318    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:24.266416 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.266344    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:24.266416 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.266362    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d730be04-38d0-48a6-8235-20372487491f-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:24.266416 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.266377    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:24.266416 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.266392    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:24.267420 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.267386    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-istio-envoy" pod "d730be04-38d0-48a6-8235-20372487491f" (UID: "d730be04-38d0-48a6-8235-20372487491f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:24.267611 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.267578    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d730be04-38d0-48a6-8235-20372487491f-istio-podinfo" pod "d730be04-38d0-48a6-8235-20372487491f" (UID: "d730be04-38d0-48a6-8235-20372487491f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:40:24.267720 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.267700    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d730be04-38d0-48a6-8235-20372487491f-istio-token" pod "d730be04-38d0-48a6-8235-20372487491f" (UID: "d730be04-38d0-48a6-8235-20372487491f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:40:24.267958 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.267937    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d730be04-38d0-48a6-8235-20372487491f-kube-api-access-rgg5q" pod "d730be04-38d0-48a6-8235-20372487491f" (UID: "d730be04-38d0-48a6-8235-20372487491f"). InnerVolumeSpecName "kube-api-access-rgg5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:40:24.370129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.370094    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d730be04-38d0-48a6-8235-20372487491f-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:24.370129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.370131    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgg5q\" (UniqueName: \"kubernetes.io/projected/d730be04-38d0-48a6-8235-20372487491f-kube-api-access-rgg5q\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:24.370368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.370154    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d730be04-38d0-48a6-8235-20372487491f-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:24.370368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.370167    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d730be04-38d0-48a6-8235-20372487491f-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:24.621821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.621794    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"]
Oct 30 23:40:24.625212 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.625191    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-7ms4r"]
Oct 30 23:40:24.748548 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.748524    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:40:24.877534 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.877507    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c26c4072-f01f-45ae-bb38-4e62b8e87617-kube-api-access-x87r5\" (UniqueName: \"kubernetes.io/projected/c26c4072-f01f-45ae-bb38-4e62b8e87617-kube-api-access-x87r5\") pod \"c26c4072-f01f-45ae-bb38-4e62b8e87617\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") "
Oct 30 23:40:24.877723 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.877677    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-token\" (UniqueName: \"kubernetes.io/projected/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-token\") pod \"c26c4072-f01f-45ae-bb38-4e62b8e87617\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") "
Oct 30 23:40:24.877802 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.877777    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c26c4072-f01f-45ae-bb38-4e62b8e87617-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26c4072-f01f-45ae-bb38-4e62b8e87617-istiod-ca-cert\") pod \"c26c4072-f01f-45ae-bb38-4e62b8e87617\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") "
Oct 30 23:40:24.877870 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.877809    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-workload-certs\") pod \"c26c4072-f01f-45ae-bb38-4e62b8e87617\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") "
Oct 30 23:40:24.877870 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.877859    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-podinfo\") pod \"c26c4072-f01f-45ae-bb38-4e62b8e87617\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") "
Oct 30 23:40:24.877983 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.877946    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-data\") pod \"c26c4072-f01f-45ae-bb38-4e62b8e87617\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") "
Oct 30 23:40:24.878039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.877994    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-workload-socket\") pod \"c26c4072-f01f-45ae-bb38-4e62b8e87617\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") "
Oct 30 23:40:24.878165 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.878088    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-credential-socket\") pod \"c26c4072-f01f-45ae-bb38-4e62b8e87617\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") "
Oct 30 23:40:24.878165 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.878138    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-envoy\") pod \"c26c4072-f01f-45ae-bb38-4e62b8e87617\" (UID: \"c26c4072-f01f-45ae-bb38-4e62b8e87617\") "
Oct 30 23:40:24.878272 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.878231    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26c4072-f01f-45ae-bb38-4e62b8e87617-istiod-ca-cert" pod "c26c4072-f01f-45ae-bb38-4e62b8e87617" (UID: "c26c4072-f01f-45ae-bb38-4e62b8e87617"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:40:24.878559 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.878493    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-data" pod "c26c4072-f01f-45ae-bb38-4e62b8e87617" (UID: "c26c4072-f01f-45ae-bb38-4e62b8e87617"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:24.878710 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.878698    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-workload-certs" pod "c26c4072-f01f-45ae-bb38-4e62b8e87617" (UID: "c26c4072-f01f-45ae-bb38-4e62b8e87617"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:24.878772 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.878754    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:24.878829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.878778    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c26c4072-f01f-45ae-bb38-4e62b8e87617-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:24.878998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.878976    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-workload-socket" pod "c26c4072-f01f-45ae-bb38-4e62b8e87617" (UID: "c26c4072-f01f-45ae-bb38-4e62b8e87617"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:24.879273 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.879246    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-credential-socket" pod "c26c4072-f01f-45ae-bb38-4e62b8e87617" (UID: "c26c4072-f01f-45ae-bb38-4e62b8e87617"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:24.880319 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.880290    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26c4072-f01f-45ae-bb38-4e62b8e87617-kube-api-access-x87r5" pod "c26c4072-f01f-45ae-bb38-4e62b8e87617" (UID: "c26c4072-f01f-45ae-bb38-4e62b8e87617"). InnerVolumeSpecName "kube-api-access-x87r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:40:24.880651 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.880623    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-token" pod "c26c4072-f01f-45ae-bb38-4e62b8e87617" (UID: "c26c4072-f01f-45ae-bb38-4e62b8e87617"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:40:24.881154 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.881129    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-podinfo" pod "c26c4072-f01f-45ae-bb38-4e62b8e87617" (UID: "c26c4072-f01f-45ae-bb38-4e62b8e87617"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:40:24.881317 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.881299    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-envoy" pod "c26c4072-f01f-45ae-bb38-4e62b8e87617" (UID: "c26c4072-f01f-45ae-bb38-4e62b8e87617"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:24.980523 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.980492    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:24.980523 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.980521    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:24.980743 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.980532    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:24.980743 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.980541    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:24.980743 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.980549    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:24.980743 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.980558    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x87r5\" (UniqueName: \"kubernetes.io/projected/c26c4072-f01f-45ae-bb38-4e62b8e87617-kube-api-access-x87r5\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:24.980743 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:24.980568    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c26c4072-f01f-45ae-bb38-4e62b8e87617-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:25.213846 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:25.213764    5078 generic.go:356] "Generic (PLEG): container finished" podID="c26c4072-f01f-45ae-bb38-4e62b8e87617" containerID="520e91a0e5734fe74c1ec55bb42716604d0e61a0b71d9b7d28edf926e08b7087" exitCode=0
Oct 30 23:40:25.213846 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:25.213810    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"
Oct 30 23:40:25.214370 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:25.213848    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch" event={"ID":"c26c4072-f01f-45ae-bb38-4e62b8e87617","Type":"ContainerDied","Data":"520e91a0e5734fe74c1ec55bb42716604d0e61a0b71d9b7d28edf926e08b7087"}
Oct 30 23:40:25.214370 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:25.213931    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch" event={"ID":"c26c4072-f01f-45ae-bb38-4e62b8e87617","Type":"ContainerDied","Data":"cdffdb62ec0919d9c410653dea6a0e08ab9d44fe68d1b5483f599938be1df0dd"}
Oct 30 23:40:25.214370 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:25.213965    5078 scope.go:122] "RemoveContainer" containerID="520e91a0e5734fe74c1ec55bb42716604d0e61a0b71d9b7d28edf926e08b7087"
Oct 30 23:40:25.225381 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:25.225366    5078 scope.go:122] "RemoveContainer" containerID="520e91a0e5734fe74c1ec55bb42716604d0e61a0b71d9b7d28edf926e08b7087"
Oct 30 23:40:25.225661 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:40:25.225642    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"520e91a0e5734fe74c1ec55bb42716604d0e61a0b71d9b7d28edf926e08b7087\": container with ID starting with 520e91a0e5734fe74c1ec55bb42716604d0e61a0b71d9b7d28edf926e08b7087 not found: ID does not exist" containerID="520e91a0e5734fe74c1ec55bb42716604d0e61a0b71d9b7d28edf926e08b7087"
Oct 30 23:40:25.225723 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:25.225670    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"520e91a0e5734fe74c1ec55bb42716604d0e61a0b71d9b7d28edf926e08b7087"} err="failed to get container status \"520e91a0e5734fe74c1ec55bb42716604d0e61a0b71d9b7d28edf926e08b7087\": rpc error: code = NotFound desc = could not find container \"520e91a0e5734fe74c1ec55bb42716604d0e61a0b71d9b7d28edf926e08b7087\": container with ID starting with 520e91a0e5734fe74c1ec55bb42716604d0e61a0b71d9b7d28edf926e08b7087 not found: ID does not exist"
Oct 30 23:40:25.379342 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:25.379308    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"]
Oct 30 23:40:25.383939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:25.383914    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-lhkch"]
Oct 30 23:40:26.120212 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.120191    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:40:26.218561 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.218489    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-workload-certs\") pod \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") "
Oct 30 23:40:26.218561 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.218541    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-envoy\") pod \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") "
Oct 30 23:40:26.219063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.218690    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istiod-ca-cert\") pod \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") "
Oct 30 23:40:26.219063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.218712    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-token\" (UniqueName: \"kubernetes.io/projected/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-token\") pod \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") "
Oct 30 23:40:26.219063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.218793    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-data\") pod \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") "
Oct 30 23:40:26.219063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.218828    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-workload-socket\") pod \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") "
Oct 30 23:40:26.219063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.218792    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-workload-certs" pod "beb7cc10-970b-4a76-9f24-9151f8c7c0c8" (UID: "beb7cc10-970b-4a76-9f24-9151f8c7c0c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:26.219063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.218985    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-podinfo\") pod \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") "
Oct 30 23:40:26.219063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.219047    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-kube-api-access-qct4t\" (UniqueName: \"kubernetes.io/projected/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-kube-api-access-qct4t\") pod \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") "
Oct 30 23:40:26.219435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.219134    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istiod-ca-cert" pod "beb7cc10-970b-4a76-9f24-9151f8c7c0c8" (UID: "beb7cc10-970b-4a76-9f24-9151f8c7c0c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:40:26.219435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.219143    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-credential-socket\") pod \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\" (UID: \"beb7cc10-970b-4a76-9f24-9151f8c7c0c8\") "
Oct 30 23:40:26.219435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.219138    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-data" pod "beb7cc10-970b-4a76-9f24-9151f8c7c0c8" (UID: "beb7cc10-970b-4a76-9f24-9151f8c7c0c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:26.219435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.219343    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-workload-socket" pod "beb7cc10-970b-4a76-9f24-9151f8c7c0c8" (UID: "beb7cc10-970b-4a76-9f24-9151f8c7c0c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:26.219435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.219403    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-credential-socket" pod "beb7cc10-970b-4a76-9f24-9151f8c7c0c8" (UID: "beb7cc10-970b-4a76-9f24-9151f8c7c0c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:26.219949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.219791    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:26.219949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.219811    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:26.219949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.219821    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:26.219949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.219830    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:26.219949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.219843    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:26.221343 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.221320    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-envoy" pod "beb7cc10-970b-4a76-9f24-9151f8c7c0c8" (UID: "beb7cc10-970b-4a76-9f24-9151f8c7c0c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:40:26.221453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.221396    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-podinfo" pod "beb7cc10-970b-4a76-9f24-9151f8c7c0c8" (UID: "beb7cc10-970b-4a76-9f24-9151f8c7c0c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:40:26.221903 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.221882    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-token" pod "beb7cc10-970b-4a76-9f24-9151f8c7c0c8" (UID: "beb7cc10-970b-4a76-9f24-9151f8c7c0c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:40:26.221979 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.221912    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-kube-api-access-qct4t" pod "beb7cc10-970b-4a76-9f24-9151f8c7c0c8" (UID: "beb7cc10-970b-4a76-9f24-9151f8c7c0c8"). InnerVolumeSpecName "kube-api-access-qct4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:40:26.225202 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.225174    5078 generic.go:356] "Generic (PLEG): container finished" podID="beb7cc10-970b-4a76-9f24-9151f8c7c0c8" containerID="020f0bbad6d3610327973300fc5470188ec173d9f55015e27dfe19e214ec4ef1" exitCode=0
Oct 30 23:40:26.225294 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.225216    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"
Oct 30 23:40:26.225294 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.225246    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg" event={"ID":"beb7cc10-970b-4a76-9f24-9151f8c7c0c8","Type":"ContainerDied","Data":"020f0bbad6d3610327973300fc5470188ec173d9f55015e27dfe19e214ec4ef1"}
Oct 30 23:40:26.225371 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.225296    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg" event={"ID":"beb7cc10-970b-4a76-9f24-9151f8c7c0c8","Type":"ContainerDied","Data":"0c0bcadbfb269e75b2cf327aaffe47420ec37446cbb95133b3bb05c7b8576cae"}
Oct 30 23:40:26.225371 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.225317    5078 scope.go:122] "RemoveContainer" containerID="020f0bbad6d3610327973300fc5470188ec173d9f55015e27dfe19e214ec4ef1"
Oct 30 23:40:26.253259 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.253239    5078 scope.go:122] "RemoveContainer" containerID="020f0bbad6d3610327973300fc5470188ec173d9f55015e27dfe19e214ec4ef1"
Oct 30 23:40:26.253633 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:40:26.253587    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"020f0bbad6d3610327973300fc5470188ec173d9f55015e27dfe19e214ec4ef1\": container with ID starting with 020f0bbad6d3610327973300fc5470188ec173d9f55015e27dfe19e214ec4ef1 not found: ID does not exist" containerID="020f0bbad6d3610327973300fc5470188ec173d9f55015e27dfe19e214ec4ef1"
Oct 30 23:40:26.253712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.253643    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"020f0bbad6d3610327973300fc5470188ec173d9f55015e27dfe19e214ec4ef1"} err="failed to get container status \"020f0bbad6d3610327973300fc5470188ec173d9f55015e27dfe19e214ec4ef1\": rpc error: code = NotFound desc = could not find container \"020f0bbad6d3610327973300fc5470188ec173d9f55015e27dfe19e214ec4ef1\": container with ID starting with 020f0bbad6d3610327973300fc5470188ec173d9f55015e27dfe19e214ec4ef1 not found: ID does not exist"
Oct 30 23:40:26.299996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.299967    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c26c4072-f01f-45ae-bb38-4e62b8e87617" path="/var/lib/kubelet/pods/c26c4072-f01f-45ae-bb38-4e62b8e87617/volumes"
Oct 30 23:40:26.300486 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.300472    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d730be04-38d0-48a6-8235-20372487491f" path="/var/lib/kubelet/pods/d730be04-38d0-48a6-8235-20372487491f/volumes"
Oct 30 23:40:26.321054 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.321031    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:26.321054 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.321053    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qct4t\" (UniqueName: \"kubernetes.io/projected/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-kube-api-access-qct4t\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:26.321206 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.321064    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:26.321206 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.321073    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/beb7cc10-970b-4a76-9f24-9151f8c7c0c8-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:26.403031 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.403001    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"]
Oct 30 23:40:26.406788 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:26.406755    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-frpdg"]
Oct 30 23:40:28.299385 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:28.299352    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beb7cc10-970b-4a76-9f24-9151f8c7c0c8" path="/var/lib/kubelet/pods/beb7cc10-970b-4a76-9f24-9151f8c7c0c8/volumes"
Oct 30 23:40:33.197976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:33.197949    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j"
Oct 30 23:40:39.036762 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.036729    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx"]
Oct 30 23:40:39.037384 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.037369    5078 cpu_manager.go:414] "containerMap: removing container" podUID="beb7cc10-970b-4a76-9f24-9151f8c7c0c8" containerName="istio-proxy"
Oct 30 23:40:39.037431 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.037386    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beb7cc10-970b-4a76-9f24-9151f8c7c0c8" containerName="istio-proxy"
Oct 30 23:40:39.037431 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.037408    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d730be04-38d0-48a6-8235-20372487491f" containerName="istio-proxy"
Oct 30 23:40:39.037431 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.037413    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d730be04-38d0-48a6-8235-20372487491f" containerName="istio-proxy"
Oct 30 23:40:39.037531 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.037435    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c26c4072-f01f-45ae-bb38-4e62b8e87617" containerName="istio-proxy"
Oct 30 23:40:39.037531 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.037443    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c26c4072-f01f-45ae-bb38-4e62b8e87617" containerName="istio-proxy"
Oct 30 23:40:39.037629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.037542    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d730be04-38d0-48a6-8235-20372487491f" containerName="istio-proxy"
Oct 30 23:40:39.037629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.037552    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c26c4072-f01f-45ae-bb38-4e62b8e87617" containerName="istio-proxy"
Oct 30 23:40:39.037629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.037563    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="beb7cc10-970b-4a76-9f24-9151f8c7c0c8" containerName="istio-proxy"
Oct 30 23:40:39.042409 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.042391    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx"
Oct 30 23:40:39.094759 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.094727    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:40:39.094759 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.094784    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:40:39.112819 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.112788    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9011644f-f041-4f08-a46e-d3795f8fd144-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx\" (UID: \"9011644f-f041-4f08-a46e-d3795f8fd144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx"
Oct 30 23:40:39.113307 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.113278    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bbml\" (UniqueName: \"kubernetes.io/projected/9011644f-f041-4f08-a46e-d3795f8fd144-kube-api-access-6bbml\") pod \"maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx\" (UID: \"9011644f-f041-4f08-a46e-d3795f8fd144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx"
Oct 30 23:40:39.113494 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.113469    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9011644f-f041-4f08-a46e-d3795f8fd144-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx\" (UID: \"9011644f-f041-4f08-a46e-d3795f8fd144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx"
Oct 30 23:40:39.115471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.115450    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:40:39.115579 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.115488    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:40:39.148994 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.148967    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx"]
Oct 30 23:40:39.214993 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.214963    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bbml\" (UniqueName: \"kubernetes.io/projected/9011644f-f041-4f08-a46e-d3795f8fd144-kube-api-access-6bbml\") pod \"maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx\" (UID: \"9011644f-f041-4f08-a46e-d3795f8fd144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx"
Oct 30 23:40:39.215130 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.215039    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9011644f-f041-4f08-a46e-d3795f8fd144-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx\" (UID: \"9011644f-f041-4f08-a46e-d3795f8fd144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx"
Oct 30 23:40:39.215130 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.215086    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9011644f-f041-4f08-a46e-d3795f8fd144-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx\" (UID: \"9011644f-f041-4f08-a46e-d3795f8fd144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx"
Oct 30 23:40:39.215708 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.215689    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9011644f-f041-4f08-a46e-d3795f8fd144-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx\" (UID: \"9011644f-f041-4f08-a46e-d3795f8fd144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx"
Oct 30 23:40:39.217696 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.217678    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9011644f-f041-4f08-a46e-d3795f8fd144-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx\" (UID: \"9011644f-f041-4f08-a46e-d3795f8fd144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx"
Oct 30 23:40:39.288672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.284742    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bbml\" (UniqueName: \"kubernetes.io/projected/9011644f-f041-4f08-a46e-d3795f8fd144-kube-api-access-6bbml\") pod \"maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx\" (UID: \"9011644f-f041-4f08-a46e-d3795f8fd144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx"
Oct 30 23:40:39.356461 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.356425    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx"
Oct 30 23:40:39.754009 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:40:39.753982    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9011644f_f041_4f08_a46e_d3795f8fd144.slice/crio-ebbf18fa7d65bf8d04f1ef6f9111d01a1bbac74c8d0d03bc47eef62f22faf92e WatchSource:0}: Error finding container ebbf18fa7d65bf8d04f1ef6f9111d01a1bbac74c8d0d03bc47eef62f22faf92e: Status 404 returned error can't find the container with id ebbf18fa7d65bf8d04f1ef6f9111d01a1bbac74c8d0d03bc47eef62f22faf92e
Oct 30 23:40:39.758339 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:39.754706    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx"]
Oct 30 23:40:40.346354 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:40.346319    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx" event={"ID":"9011644f-f041-4f08-a46e-d3795f8fd144","Type":"ContainerStarted","Data":"40a3ed4598a409cbf1498564b0adc1676d7d86fe9eec3314e37b49427fb81af0"}
Oct 30 23:40:40.346791 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:40.346367    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx" event={"ID":"9011644f-f041-4f08-a46e-d3795f8fd144","Type":"ContainerStarted","Data":"ebbf18fa7d65bf8d04f1ef6f9111d01a1bbac74c8d0d03bc47eef62f22faf92e"}
Oct 30 23:40:40.460099 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:40.460052    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx" podStartSLOduration=2.46003698 podStartE2EDuration="2.46003698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:40.458189326 +0000 UTC m=+2026.346095691" watchObservedRunningTime="2026-10-30 23:40:40.46003698 +0000 UTC m=+2026.347943340"
Oct 30 23:40:41.355343 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:41.355305    5078 generic.go:356] "Generic (PLEG): container finished" podID="9011644f-f041-4f08-a46e-d3795f8fd144" containerID="40a3ed4598a409cbf1498564b0adc1676d7d86fe9eec3314e37b49427fb81af0" exitCode=0
Oct 30 23:40:41.355810 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:41.355384    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx" event={"ID":"9011644f-f041-4f08-a46e-d3795f8fd144","Type":"ContainerDied","Data":"40a3ed4598a409cbf1498564b0adc1676d7d86fe9eec3314e37b49427fb81af0"}
Oct 30 23:40:41.376746 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:41.376716    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd"]
Oct 30 23:40:41.425884 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:41.425852    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-4qzjd"]
Oct 30 23:40:42.300420 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:42.300379    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4ccbc2f-d688-471f-bef0-60d3f6bd456b" path="/var/lib/kubelet/pods/d4ccbc2f-d688-471f-bef0-60d3f6bd456b/volumes"
Oct 30 23:40:42.530005 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:42.529978    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx"
Oct 30 23:40:42.648653 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:42.648574    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9011644f-f041-4f08-a46e-d3795f8fd144-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9011644f-f041-4f08-a46e-d3795f8fd144-maas-api-openshift-ca\") pod \"9011644f-f041-4f08-a46e-d3795f8fd144\" (UID: \"9011644f-f041-4f08-a46e-d3795f8fd144\") "
Oct 30 23:40:42.648828 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:42.648678    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9011644f-f041-4f08-a46e-d3795f8fd144-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9011644f-f041-4f08-a46e-d3795f8fd144-maas-api-xks-ca\") pod \"9011644f-f041-4f08-a46e-d3795f8fd144\" (UID: \"9011644f-f041-4f08-a46e-d3795f8fd144\") "
Oct 30 23:40:42.648828 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:42.648797    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9011644f-f041-4f08-a46e-d3795f8fd144-kube-api-access-6bbml\" (UniqueName: \"kubernetes.io/projected/9011644f-f041-4f08-a46e-d3795f8fd144-kube-api-access-6bbml\") pod \"9011644f-f041-4f08-a46e-d3795f8fd144\" (UID: \"9011644f-f041-4f08-a46e-d3795f8fd144\") "
Oct 30 23:40:42.648962 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:42.648937    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9011644f-f041-4f08-a46e-d3795f8fd144-maas-api-openshift-ca" pod "9011644f-f041-4f08-a46e-d3795f8fd144" (UID: "9011644f-f041-4f08-a46e-d3795f8fd144"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:40:42.649502 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:42.649468    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9011644f-f041-4f08-a46e-d3795f8fd144-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:42.651124 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:42.651095    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9011644f-f041-4f08-a46e-d3795f8fd144-kube-api-access-6bbml" pod "9011644f-f041-4f08-a46e-d3795f8fd144" (UID: "9011644f-f041-4f08-a46e-d3795f8fd144"). InnerVolumeSpecName "kube-api-access-6bbml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:40:42.651235 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:42.651199    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9011644f-f041-4f08-a46e-d3795f8fd144-maas-api-xks-ca" pod "9011644f-f041-4f08-a46e-d3795f8fd144" (UID: "9011644f-f041-4f08-a46e-d3795f8fd144"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:40:42.750340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:42.750304    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bbml\" (UniqueName: \"kubernetes.io/projected/9011644f-f041-4f08-a46e-d3795f8fd144-kube-api-access-6bbml\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:42.750340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:42.750339    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9011644f-f041-4f08-a46e-d3795f8fd144-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:40:43.371942 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:43.371896    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx" event={"ID":"9011644f-f041-4f08-a46e-d3795f8fd144","Type":"ContainerDied","Data":"ebbf18fa7d65bf8d04f1ef6f9111d01a1bbac74c8d0d03bc47eef62f22faf92e"}
Oct 30 23:40:43.372122 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:43.371959    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebbf18fa7d65bf8d04f1ef6f9111d01a1bbac74c8d0d03bc47eef62f22faf92e"
Oct 30 23:40:43.372122 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:43.371913    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx"
Oct 30 23:40:47.593512 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:40:47.593478    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:40:53.220629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:53.219651    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs"]
Oct 30 23:40:53.221164 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:53.221068    5078 cpu_manager.go:414] "containerMap: removing container" podUID="9011644f-f041-4f08-a46e-d3795f8fd144" containerName="revoke-keys"
Oct 30 23:40:53.221164 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:53.221089    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9011644f-f041-4f08-a46e-d3795f8fd144" containerName="revoke-keys"
Oct 30 23:40:53.221293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:53.221245    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="9011644f-f041-4f08-a46e-d3795f8fd144" containerName="revoke-keys"
Oct 30 23:40:53.256951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:53.256914    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs"
Oct 30 23:40:53.296548 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:53.296518    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs"]
Oct 30 23:40:53.343017 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:53.342983    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-55666687-dockercfg-ccbfh\""
Oct 30 23:40:53.346626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:53.343410    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-55666687\""
Oct 30 23:40:53.428687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:53.428651    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0307507-cd4c-4129-96ec-9005d468f73b-plugins-config-volume\") pod \"payload-processing-e2e-mt-55666687-cdc987d69-km7zs\" (UID: \"e0307507-cd4c-4129-96ec-9005d468f73b\") " pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs"
Oct 30 23:40:53.429100 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:53.429072    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmskv\" (UniqueName: \"kubernetes.io/projected/e0307507-cd4c-4129-96ec-9005d468f73b-kube-api-access-kmskv\") pod \"payload-processing-e2e-mt-55666687-cdc987d69-km7zs\" (UID: \"e0307507-cd4c-4129-96ec-9005d468f73b\") " pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs"
Oct 30 23:40:53.530373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:53.530290    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmskv\" (UniqueName: \"kubernetes.io/projected/e0307507-cd4c-4129-96ec-9005d468f73b-kube-api-access-kmskv\") pod \"payload-processing-e2e-mt-55666687-cdc987d69-km7zs\" (UID: \"e0307507-cd4c-4129-96ec-9005d468f73b\") " pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs"
Oct 30 23:40:53.530559 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:53.530475    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0307507-cd4c-4129-96ec-9005d468f73b-plugins-config-volume\") pod \"payload-processing-e2e-mt-55666687-cdc987d69-km7zs\" (UID: \"e0307507-cd4c-4129-96ec-9005d468f73b\") " pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs"
Oct 30 23:40:53.531210 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:53.531190    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0307507-cd4c-4129-96ec-9005d468f73b-plugins-config-volume\") pod \"payload-processing-e2e-mt-55666687-cdc987d69-km7zs\" (UID: \"e0307507-cd4c-4129-96ec-9005d468f73b\") " pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs"
Oct 30 23:40:53.586942 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:53.586914    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmskv\" (UniqueName: \"kubernetes.io/projected/e0307507-cd4c-4129-96ec-9005d468f73b-kube-api-access-kmskv\") pod \"payload-processing-e2e-mt-55666687-cdc987d69-km7zs\" (UID: \"e0307507-cd4c-4129-96ec-9005d468f73b\") " pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs"
Oct 30 23:40:53.626589 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:53.626555    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs"
Oct 30 23:40:54.007542 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:54.007518    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs"]
Oct 30 23:40:54.008464 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:40:54.008443    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0307507_cd4c_4129_96ec_9005d468f73b.slice/crio-0f45bdf37c9686bc5f18df41be79932e65442bdcffdcd622ea124172ca000a3c WatchSource:0}: Error finding container 0f45bdf37c9686bc5f18df41be79932e65442bdcffdcd622ea124172ca000a3c: Status 404 returned error can't find the container with id 0f45bdf37c9686bc5f18df41be79932e65442bdcffdcd622ea124172ca000a3c
Oct 30 23:40:54.481252 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:54.481213    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs" event={"ID":"e0307507-cd4c-4129-96ec-9005d468f73b","Type":"ContainerStarted","Data":"ea956851a9c2473d70780cffece5dc617645f4927f2a00b3e8e93f458419ab18"}
Oct 30 23:40:54.481742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:54.481266    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs" event={"ID":"e0307507-cd4c-4129-96ec-9005d468f73b","Type":"ContainerStarted","Data":"0f45bdf37c9686bc5f18df41be79932e65442bdcffdcd622ea124172ca000a3c"}
Oct 30 23:40:54.481742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:54.481406    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs"
Oct 30 23:40:54.595621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:54.595063    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs" podStartSLOduration=1.595040923 podStartE2EDuration="1.595040923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:40:54.594215515 +0000 UTC m=+2040.482121868" watchObservedRunningTime="2026-10-30 23:40:54.595040923 +0000 UTC m=+2040.482947277"
Oct 30 23:40:58.045863 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:58.045827    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf"]
Oct 30 23:40:58.049270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:58.049244    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc285e6e-qndpf"]
Oct 30 23:40:58.302658 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:40:58.302565    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88c35deb-c0ef-4790-ac5b-725649bba7b5" path="/var/lib/kubelet/pods/88c35deb-c0ef-4790-ac5b-725649bba7b5/volumes"
Oct 30 23:41:01.659802 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:01.659765    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh"]
Oct 30 23:41:01.665417 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:01.665394    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh"
Oct 30 23:41:01.719676 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:01.719649    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh"]
Oct 30 23:41:01.764715 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:01.764684    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:41:01.764715 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:01.764708    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:41:01.764917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:01.764684    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:41:01.766919 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:01.766906    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:41:01.797183 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:01.797156    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bcca03c-24ce-47a1-9bfc-225f2818d17c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh\" (UID: \"3bcca03c-24ce-47a1-9bfc-225f2818d17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh"
Oct 30 23:41:01.797335 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:01.797214    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bltdp\" (UniqueName: \"kubernetes.io/projected/3bcca03c-24ce-47a1-9bfc-225f2818d17c-kube-api-access-bltdp\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh\" (UID: \"3bcca03c-24ce-47a1-9bfc-225f2818d17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh"
Oct 30 23:41:01.797488 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:01.797470    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bcca03c-24ce-47a1-9bfc-225f2818d17c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh\" (UID: \"3bcca03c-24ce-47a1-9bfc-225f2818d17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh"
Oct 30 23:41:01.898579 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:01.898541    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bcca03c-24ce-47a1-9bfc-225f2818d17c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh\" (UID: \"3bcca03c-24ce-47a1-9bfc-225f2818d17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh"
Oct 30 23:41:01.898805 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:01.898682    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bcca03c-24ce-47a1-9bfc-225f2818d17c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh\" (UID: \"3bcca03c-24ce-47a1-9bfc-225f2818d17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh"
Oct 30 23:41:01.898805 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:01.898746    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bltdp\" (UniqueName: \"kubernetes.io/projected/3bcca03c-24ce-47a1-9bfc-225f2818d17c-kube-api-access-bltdp\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh\" (UID: \"3bcca03c-24ce-47a1-9bfc-225f2818d17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh"
Oct 30 23:41:01.899279 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:01.899258    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bcca03c-24ce-47a1-9bfc-225f2818d17c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh\" (UID: \"3bcca03c-24ce-47a1-9bfc-225f2818d17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh"
Oct 30 23:41:01.901225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:01.901198    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bcca03c-24ce-47a1-9bfc-225f2818d17c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh\" (UID: \"3bcca03c-24ce-47a1-9bfc-225f2818d17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh"
Oct 30 23:41:01.955652 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:01.955538    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bltdp\" (UniqueName: \"kubernetes.io/projected/3bcca03c-24ce-47a1-9bfc-225f2818d17c-kube-api-access-bltdp\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh\" (UID: \"3bcca03c-24ce-47a1-9bfc-225f2818d17c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh"
Oct 30 23:41:01.979698 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:01.979668    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh"
Oct 30 23:41:02.358750 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:02.358714    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh"]
Oct 30 23:41:02.358999 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:41:02.358976    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bcca03c_24ce_47a1_9bfc_225f2818d17c.slice/crio-9805dede5cbf04356449e61a7ee89e49850aa1253e042d3ab406f1592c677137 WatchSource:0}: Error finding container 9805dede5cbf04356449e61a7ee89e49850aa1253e042d3ab406f1592c677137: Status 404 returned error can't find the container with id 9805dede5cbf04356449e61a7ee89e49850aa1253e042d3ab406f1592c677137
Oct 30 23:41:02.651487 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:02.651412    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh" event={"ID":"3bcca03c-24ce-47a1-9bfc-225f2818d17c","Type":"ContainerStarted","Data":"9805dede5cbf04356449e61a7ee89e49850aa1253e042d3ab406f1592c677137"}
Oct 30 23:41:02.785335 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:02.785309    5078 scope.go:122] "RemoveContainer" containerID="35c5213216ed5f0548b4747bf4fde695499bfe2571090d6b09816c8108dc9a82"
Oct 30 23:41:02.794689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:02.794576    5078 scope.go:122] "RemoveContainer" containerID="1b2addfc5cc4a1f87a54d7e79a81c8fd4209b8539a4283b8fc32b9bdf8532655"
Oct 30 23:41:03.664831 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:03.664749    5078 generic.go:356] "Generic (PLEG): container finished" podID="3bcca03c-24ce-47a1-9bfc-225f2818d17c" containerID="e6f46ca9d662f8d58e031ff5740b06d47bb3cd6dfcc931a270d04b12b39a4465" exitCode=0
Oct 30 23:41:03.664962 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:03.664818    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh" event={"ID":"3bcca03c-24ce-47a1-9bfc-225f2818d17c","Type":"ContainerDied","Data":"e6f46ca9d662f8d58e031ff5740b06d47bb3cd6dfcc931a270d04b12b39a4465"}
Oct 30 23:41:04.839895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:04.839869    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh"
Oct 30 23:41:04.946075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:04.945987    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bcca03c-24ce-47a1-9bfc-225f2818d17c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bcca03c-24ce-47a1-9bfc-225f2818d17c-maas-api-xks-ca\") pod \"3bcca03c-24ce-47a1-9bfc-225f2818d17c\" (UID: \"3bcca03c-24ce-47a1-9bfc-225f2818d17c\") "
Oct 30 23:41:04.946247 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:04.946178    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bcca03c-24ce-47a1-9bfc-225f2818d17c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bcca03c-24ce-47a1-9bfc-225f2818d17c-maas-api-openshift-ca\") pod \"3bcca03c-24ce-47a1-9bfc-225f2818d17c\" (UID: \"3bcca03c-24ce-47a1-9bfc-225f2818d17c\") "
Oct 30 23:41:04.946366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:04.946351    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bcca03c-24ce-47a1-9bfc-225f2818d17c-kube-api-access-bltdp\" (UniqueName: \"kubernetes.io/projected/3bcca03c-24ce-47a1-9bfc-225f2818d17c-kube-api-access-bltdp\") pod \"3bcca03c-24ce-47a1-9bfc-225f2818d17c\" (UID: \"3bcca03c-24ce-47a1-9bfc-225f2818d17c\") "
Oct 30 23:41:04.946565 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:04.946525    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bcca03c-24ce-47a1-9bfc-225f2818d17c-maas-api-openshift-ca" pod "3bcca03c-24ce-47a1-9bfc-225f2818d17c" (UID: "3bcca03c-24ce-47a1-9bfc-225f2818d17c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:41:04.946968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:04.946948    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bcca03c-24ce-47a1-9bfc-225f2818d17c-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:41:04.948488 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:04.948460    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bcca03c-24ce-47a1-9bfc-225f2818d17c-maas-api-xks-ca" pod "3bcca03c-24ce-47a1-9bfc-225f2818d17c" (UID: "3bcca03c-24ce-47a1-9bfc-225f2818d17c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:41:04.948669 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:04.948651    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bcca03c-24ce-47a1-9bfc-225f2818d17c-kube-api-access-bltdp" pod "3bcca03c-24ce-47a1-9bfc-225f2818d17c" (UID: "3bcca03c-24ce-47a1-9bfc-225f2818d17c"). InnerVolumeSpecName "kube-api-access-bltdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:41:05.048845 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:05.048808    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bltdp\" (UniqueName: \"kubernetes.io/projected/3bcca03c-24ce-47a1-9bfc-225f2818d17c-kube-api-access-bltdp\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:41:05.048845 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:05.048837    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bcca03c-24ce-47a1-9bfc-225f2818d17c-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:41:05.578962 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:05.578934    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs"
Oct 30 23:41:05.679763 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:05.679729    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh" event={"ID":"3bcca03c-24ce-47a1-9bfc-225f2818d17c","Type":"ContainerDied","Data":"9805dede5cbf04356449e61a7ee89e49850aa1253e042d3ab406f1592c677137"}
Oct 30 23:41:05.679935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:05.679784    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9805dede5cbf04356449e61a7ee89e49850aa1253e042d3ab406f1592c677137"
Oct 30 23:41:05.679935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:41:05.679738    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh"
Oct 30 23:41:17.596925 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:41:17.596889    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:41:47.596752 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:41:47.596660    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:42:02.851168 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:02.851134    5078 scope.go:122] "RemoveContainer" containerID="56196ad0921e1f9edbd0c17112f0e8d9ac08e167e3670541f81a4ade7d4c403b"
Oct 30 23:42:11.788653 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:11.788618    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx"]
Oct 30 23:42:11.837368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:11.837330    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b526ac-khcbx"]
Oct 30 23:42:12.300692 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:12.300653    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9011644f-f041-4f08-a46e-d3795f8fd144" path="/var/lib/kubelet/pods/9011644f-f041-4f08-a46e-d3795f8fd144/volumes"
Oct 30 23:42:13.233363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:13.233331    5078 generic.go:356] "Generic (PLEG): container finished" podID="ae2a127b-3a86-45d1-8e4a-10557a96a711" containerID="aa59d354ec5d91a535b2c4bafd7d0a0c257a72098eb79864d3f9d74a7fda6edd" exitCode=0
Oct 30 23:42:13.233959 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:13.233400    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6" event={"ID":"ae2a127b-3a86-45d1-8e4a-10557a96a711","Type":"ContainerDied","Data":"aa59d354ec5d91a535b2c4bafd7d0a0c257a72098eb79864d3f9d74a7fda6edd"}
Oct 30 23:42:13.233959 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:13.233859    5078 scope.go:122] "RemoveContainer" containerID="aa59d354ec5d91a535b2c4bafd7d0a0c257a72098eb79864d3f9d74a7fda6edd"
Oct 30 23:42:13.235830 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:13.235798    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:42:13.235968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:13.235888    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:42:13.235968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:13.235936    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:42:14.242666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:14.242629    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6" event={"ID":"ae2a127b-3a86-45d1-8e4a-10557a96a711","Type":"ContainerStarted","Data":"21d276b2ca00ef3304604872f3206f84b51c0b3d12e0be882544e5757d26bcc9"}
Oct 30 23:42:15.045296 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:15.045259    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:42:15.046373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:15.046347    5078 patch_prober.go:29] interesting pod/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 23:42:15.046525 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:15.046402    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6" podUID="ae2a127b-3a86-45d1-8e4a-10557a96a711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 23:42:16.045197 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:16.045167    5078 patch_prober.go:29] interesting pod/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 23:42:16.045665 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:16.045232    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6" podUID="ae2a127b-3a86-45d1-8e4a-10557a96a711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 23:42:17.045642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:17.045585    5078 patch_prober.go:29] interesting pod/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 23:42:17.046108 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:17.045660    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6" podUID="ae2a127b-3a86-45d1-8e4a-10557a96a711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 23:42:17.508042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:17.507967    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:42:17.612322 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:42:17.612288    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:42:18.045203 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:18.045169    5078 patch_prober.go:29] interesting pod/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 23:42:18.045377 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:18.045225    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6" podUID="ae2a127b-3a86-45d1-8e4a-10557a96a711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 23:42:18.300058 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:18.299981    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"]
Oct 30 23:42:18.300505 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:18.300159    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8" podUID="e0f4988d-873c-469e-80b6-652ab2cadb2b" containerName="istio-proxy" containerID="cri-o://eeffbd0004e46903c0072b3f7fcc01bd3b64fe9e5ea18e64bf95f2b3b809105b" gracePeriod=30
Oct 30 23:42:19.046298 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:19.046259    5078 patch_prober.go:29] interesting pod/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 23:42:19.046489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:19.046326    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6" podUID="ae2a127b-3a86-45d1-8e4a-10557a96a711" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 23:42:19.475649 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:19.474426    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"]
Oct 30 23:42:19.475649 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:19.474748    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg" podUID="bca2f196-2daf-4071-b366-596e7ff0f23d" containerName="istio-proxy" containerID="cri-o://b79fe0f55cf57f69637487be5c0dd6d37ed5762dc3ea8a751ec0276f48ef2df2" gracePeriod=30
Oct 30 23:42:20.050742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:20.050713    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:42:20.051843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:20.051826    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:42:22.181817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.181775    5078 patch_prober.go:29] interesting pod/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:42:22.182412 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.182379    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg" podUID="bca2f196-2daf-4071-b366-596e7ff0f23d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:42:22.349859 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.349469    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg"]
Oct 30 23:42:22.350610 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.350526    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3bcca03c-24ce-47a1-9bfc-225f2818d17c" containerName="revoke-keys"
Oct 30 23:42:22.350828 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.350811    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bcca03c-24ce-47a1-9bfc-225f2818d17c" containerName="revoke-keys"
Oct 30 23:42:22.351188 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.351170    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bcca03c-24ce-47a1-9bfc-225f2818d17c" containerName="revoke-keys"
Oct 30 23:42:22.355570 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.355552    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg"
Oct 30 23:42:22.370930 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.370904    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg"]
Oct 30 23:42:22.407558 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.407519    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:42:22.407714 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.407695    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:42:22.414455 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.414422    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:42:22.414455 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.414422    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:42:22.420130 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.420108    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c46b1e-954c-4c1d-a709-6044a85b855a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg\" (UID: \"29c46b1e-954c-4c1d-a709-6044a85b855a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg"
Oct 30 23:42:22.420289 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.420266    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrfkf\" (UniqueName: \"kubernetes.io/projected/29c46b1e-954c-4c1d-a709-6044a85b855a-kube-api-access-hrfkf\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg\" (UID: \"29c46b1e-954c-4c1d-a709-6044a85b855a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg"
Oct 30 23:42:22.420585 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.420555    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c46b1e-954c-4c1d-a709-6044a85b855a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg\" (UID: \"29c46b1e-954c-4c1d-a709-6044a85b855a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg"
Oct 30 23:42:22.522636 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.522546    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrfkf\" (UniqueName: \"kubernetes.io/projected/29c46b1e-954c-4c1d-a709-6044a85b855a-kube-api-access-hrfkf\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg\" (UID: \"29c46b1e-954c-4c1d-a709-6044a85b855a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg"
Oct 30 23:42:22.522776 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.522685    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c46b1e-954c-4c1d-a709-6044a85b855a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg\" (UID: \"29c46b1e-954c-4c1d-a709-6044a85b855a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg"
Oct 30 23:42:22.522858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.522781    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c46b1e-954c-4c1d-a709-6044a85b855a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg\" (UID: \"29c46b1e-954c-4c1d-a709-6044a85b855a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg"
Oct 30 23:42:22.523439 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.523418    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c46b1e-954c-4c1d-a709-6044a85b855a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg\" (UID: \"29c46b1e-954c-4c1d-a709-6044a85b855a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg"
Oct 30 23:42:22.525189 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.525169    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c46b1e-954c-4c1d-a709-6044a85b855a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg\" (UID: \"29c46b1e-954c-4c1d-a709-6044a85b855a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg"
Oct 30 23:42:22.579274 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.579252    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrfkf\" (UniqueName: \"kubernetes.io/projected/29c46b1e-954c-4c1d-a709-6044a85b855a-kube-api-access-hrfkf\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg\" (UID: \"29c46b1e-954c-4c1d-a709-6044a85b855a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg"
Oct 30 23:42:22.672171 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.672148    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg"
Oct 30 23:42:22.907236 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.907200    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-xg4d8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:42:22.907419 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:22.907264    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8" podUID="e0f4988d-873c-469e-80b6-652ab2cadb2b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:42:23.051114 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:42:23.051078    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c46b1e_954c_4c1d_a709_6044a85b855a.slice/crio-6720e7d5b192759f7cb55bd8ef2117438c567d487bd27cc68be277098748ab93 WatchSource:0}: Error finding container 6720e7d5b192759f7cb55bd8ef2117438c567d487bd27cc68be277098748ab93: Status 404 returned error can't find the container with id 6720e7d5b192759f7cb55bd8ef2117438c567d487bd27cc68be277098748ab93
Oct 30 23:42:23.053917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.053891    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg"]
Oct 30 23:42:23.359220 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.356149    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh"]
Oct 30 23:42:23.364109 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.364085    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557e9b-4mlqh"]
Oct 30 23:42:23.370002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.369971    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg" event={"ID":"29c46b1e-954c-4c1d-a709-6044a85b855a","Type":"ContainerStarted","Data":"6720e7d5b192759f7cb55bd8ef2117438c567d487bd27cc68be277098748ab93"}
Oct 30 23:42:23.587521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.587498    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:42:23.649674 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.649641    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-credential-socket\") pod \"e0f4988d-873c-469e-80b6-652ab2cadb2b\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") "
Oct 30 23:42:23.649798 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.649677    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0f4988d-873c-469e-80b6-652ab2cadb2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f4988d-873c-469e-80b6-652ab2cadb2b-istiod-ca-cert\") pod \"e0f4988d-873c-469e-80b6-652ab2cadb2b\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") "
Oct 30 23:42:23.649798 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.649771    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f4988d-873c-469e-80b6-652ab2cadb2b-kube-api-access-l4q8r\" (UniqueName: \"kubernetes.io/projected/e0f4988d-873c-469e-80b6-652ab2cadb2b-kube-api-access-l4q8r\") pod \"e0f4988d-873c-469e-80b6-652ab2cadb2b\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") "
Oct 30 23:42:23.649897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.649819    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-workload-certs\") pod \"e0f4988d-873c-469e-80b6-652ab2cadb2b\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") "
Oct 30 23:42:23.649897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.649855    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-envoy\") pod \"e0f4988d-873c-469e-80b6-652ab2cadb2b\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") "
Oct 30 23:42:23.650005 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.649917    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-credential-socket" pod "e0f4988d-873c-469e-80b6-652ab2cadb2b" (UID: "e0f4988d-873c-469e-80b6-652ab2cadb2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:23.650084 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.650066    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-data\") pod \"e0f4988d-873c-469e-80b6-652ab2cadb2b\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") "
Oct 30 23:42:23.650154 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.650085    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0f4988d-873c-469e-80b6-652ab2cadb2b-istiod-ca-cert" pod "e0f4988d-873c-469e-80b6-652ab2cadb2b" (UID: "e0f4988d-873c-469e-80b6-652ab2cadb2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:42:23.650154 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.650092    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-workload-certs" pod "e0f4988d-873c-469e-80b6-652ab2cadb2b" (UID: "e0f4988d-873c-469e-80b6-652ab2cadb2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:23.650154 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.650126    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-podinfo\") pod \"e0f4988d-873c-469e-80b6-652ab2cadb2b\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") "
Oct 30 23:42:23.650321 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.650156    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-workload-socket\") pod \"e0f4988d-873c-469e-80b6-652ab2cadb2b\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") "
Oct 30 23:42:23.650321 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.650282    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-token\" (UniqueName: \"kubernetes.io/projected/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-token\") pod \"e0f4988d-873c-469e-80b6-652ab2cadb2b\" (UID: \"e0f4988d-873c-469e-80b6-652ab2cadb2b\") "
Oct 30 23:42:23.650321 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.650302    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-data" pod "e0f4988d-873c-469e-80b6-652ab2cadb2b" (UID: "e0f4988d-873c-469e-80b6-652ab2cadb2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:23.650469 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.650407    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-workload-socket" pod "e0f4988d-873c-469e-80b6-652ab2cadb2b" (UID: "e0f4988d-873c-469e-80b6-652ab2cadb2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:23.650923 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.650896    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:23.651048 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.650925    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0f4988d-873c-469e-80b6-652ab2cadb2b-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:23.651048 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.650941    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:23.651048 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.650954    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:23.651048 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.650966    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:23.652333 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.652307    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-envoy" pod "e0f4988d-873c-469e-80b6-652ab2cadb2b" (UID: "e0f4988d-873c-469e-80b6-652ab2cadb2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:23.652434 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.652347    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-podinfo" pod "e0f4988d-873c-469e-80b6-652ab2cadb2b" (UID: "e0f4988d-873c-469e-80b6-652ab2cadb2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:42:23.652634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.652588    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f4988d-873c-469e-80b6-652ab2cadb2b-kube-api-access-l4q8r" pod "e0f4988d-873c-469e-80b6-652ab2cadb2b" (UID: "e0f4988d-873c-469e-80b6-652ab2cadb2b"). InnerVolumeSpecName "kube-api-access-l4q8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:23.652748 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.652727    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-token" pod "e0f4988d-873c-469e-80b6-652ab2cadb2b" (UID: "e0f4988d-873c-469e-80b6-652ab2cadb2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:23.752136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.752103    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4q8r\" (UniqueName: \"kubernetes.io/projected/e0f4988d-873c-469e-80b6-652ab2cadb2b-kube-api-access-l4q8r\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:23.752136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.752129    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:23.752136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.752141    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:23.752351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:23.752150    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0f4988d-873c-469e-80b6-652ab2cadb2b-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:24.301022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.300989    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bcca03c-24ce-47a1-9bfc-225f2818d17c" path="/var/lib/kubelet/pods/3bcca03c-24ce-47a1-9bfc-225f2818d17c/volumes"
Oct 30 23:42:24.378996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.378967    5078 generic.go:356] "Generic (PLEG): container finished" podID="29c46b1e-954c-4c1d-a709-6044a85b855a" containerID="c71fdcb6205e2922f02e1e720182801068ccd928a2008cf3489899ede16bb683" exitCode=0
Oct 30 23:42:24.379442 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.379046    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg" event={"ID":"29c46b1e-954c-4c1d-a709-6044a85b855a","Type":"ContainerDied","Data":"c71fdcb6205e2922f02e1e720182801068ccd928a2008cf3489899ede16bb683"}
Oct 30 23:42:24.380647 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.380625    5078 generic.go:356] "Generic (PLEG): container finished" podID="e0f4988d-873c-469e-80b6-652ab2cadb2b" containerID="eeffbd0004e46903c0072b3f7fcc01bd3b64fe9e5ea18e64bf95f2b3b809105b" exitCode=0
Oct 30 23:42:24.380756 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.380672    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8" event={"ID":"e0f4988d-873c-469e-80b6-652ab2cadb2b","Type":"ContainerDied","Data":"eeffbd0004e46903c0072b3f7fcc01bd3b64fe9e5ea18e64bf95f2b3b809105b"}
Oct 30 23:42:24.380756 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.380680    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"
Oct 30 23:42:24.380756 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.380714    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8" event={"ID":"e0f4988d-873c-469e-80b6-652ab2cadb2b","Type":"ContainerDied","Data":"e688778a10fd884f0ed3159b72da4e6b852244514269fd1a0658361ba3010194"}
Oct 30 23:42:24.380756 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.380735    5078 scope.go:122] "RemoveContainer" containerID="eeffbd0004e46903c0072b3f7fcc01bd3b64fe9e5ea18e64bf95f2b3b809105b"
Oct 30 23:42:24.392209 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.392190    5078 scope.go:122] "RemoveContainer" containerID="eeffbd0004e46903c0072b3f7fcc01bd3b64fe9e5ea18e64bf95f2b3b809105b"
Oct 30 23:42:24.392515 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:42:24.392495    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eeffbd0004e46903c0072b3f7fcc01bd3b64fe9e5ea18e64bf95f2b3b809105b\": container with ID starting with eeffbd0004e46903c0072b3f7fcc01bd3b64fe9e5ea18e64bf95f2b3b809105b not found: ID does not exist" containerID="eeffbd0004e46903c0072b3f7fcc01bd3b64fe9e5ea18e64bf95f2b3b809105b"
Oct 30 23:42:24.392648 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.392522    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eeffbd0004e46903c0072b3f7fcc01bd3b64fe9e5ea18e64bf95f2b3b809105b"} err="failed to get container status \"eeffbd0004e46903c0072b3f7fcc01bd3b64fe9e5ea18e64bf95f2b3b809105b\": rpc error: code = NotFound desc = could not find container \"eeffbd0004e46903c0072b3f7fcc01bd3b64fe9e5ea18e64bf95f2b3b809105b\": container with ID starting with eeffbd0004e46903c0072b3f7fcc01bd3b64fe9e5ea18e64bf95f2b3b809105b not found: ID does not exist"
Oct 30 23:42:24.610440 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.610409    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"]
Oct 30 23:42:24.614202 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.614179    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-xg4d8"]
Oct 30 23:42:24.779930 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.779905    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:42:24.868521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.868454    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-workload-socket\") pod \"bca2f196-2daf-4071-b366-596e7ff0f23d\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") "
Oct 30 23:42:24.868712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.868654    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-envoy\") pod \"bca2f196-2daf-4071-b366-596e7ff0f23d\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") "
Oct 30 23:42:24.868712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.868675    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-workload-certs\") pod \"bca2f196-2daf-4071-b366-596e7ff0f23d\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") "
Oct 30 23:42:24.868712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.868711    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bca2f196-2daf-4071-b366-596e7ff0f23d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bca2f196-2daf-4071-b366-596e7ff0f23d-istiod-ca-cert\") pod \"bca2f196-2daf-4071-b366-596e7ff0f23d\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") "
Oct 30 23:42:24.868901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.868747    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-workload-socket" pod "bca2f196-2daf-4071-b366-596e7ff0f23d" (UID: "bca2f196-2daf-4071-b366-596e7ff0f23d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:24.868901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.868759    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-podinfo\") pod \"bca2f196-2daf-4071-b366-596e7ff0f23d\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") "
Oct 30 23:42:24.868901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.868818    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bca2f196-2daf-4071-b366-596e7ff0f23d-kube-api-access-xsnkv\" (UniqueName: \"kubernetes.io/projected/bca2f196-2daf-4071-b366-596e7ff0f23d-kube-api-access-xsnkv\") pod \"bca2f196-2daf-4071-b366-596e7ff0f23d\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") "
Oct 30 23:42:24.869058 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.868934    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-data\") pod \"bca2f196-2daf-4071-b366-596e7ff0f23d\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") "
Oct 30 23:42:24.869058 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.868968    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-credential-socket\") pod \"bca2f196-2daf-4071-b366-596e7ff0f23d\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") "
Oct 30 23:42:24.869058 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.868966    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-workload-certs" pod "bca2f196-2daf-4071-b366-596e7ff0f23d" (UID: "bca2f196-2daf-4071-b366-596e7ff0f23d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:24.869058 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.869042    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-token\" (UniqueName: \"kubernetes.io/projected/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-token\") pod \"bca2f196-2daf-4071-b366-596e7ff0f23d\" (UID: \"bca2f196-2daf-4071-b366-596e7ff0f23d\") "
Oct 30 23:42:24.869266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.869236    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-credential-socket" pod "bca2f196-2daf-4071-b366-596e7ff0f23d" (UID: "bca2f196-2daf-4071-b366-596e7ff0f23d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:24.869325 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.869262    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-data" pod "bca2f196-2daf-4071-b366-596e7ff0f23d" (UID: "bca2f196-2daf-4071-b366-596e7ff0f23d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:24.869382 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.869341    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca2f196-2daf-4071-b366-596e7ff0f23d-istiod-ca-cert" pod "bca2f196-2daf-4071-b366-596e7ff0f23d" (UID: "bca2f196-2daf-4071-b366-596e7ff0f23d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:42:24.869757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.869738    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:24.869844 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.869761    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bca2f196-2daf-4071-b366-596e7ff0f23d-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:24.869844 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.869775    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:24.869844 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.869789    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:24.869844 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.869803    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:24.871042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.871015    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-envoy" pod "bca2f196-2daf-4071-b366-596e7ff0f23d" (UID: "bca2f196-2daf-4071-b366-596e7ff0f23d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:24.871524 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.871504    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-podinfo" pod "bca2f196-2daf-4071-b366-596e7ff0f23d" (UID: "bca2f196-2daf-4071-b366-596e7ff0f23d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:42:24.871637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.871561    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca2f196-2daf-4071-b366-596e7ff0f23d-kube-api-access-xsnkv" pod "bca2f196-2daf-4071-b366-596e7ff0f23d" (UID: "bca2f196-2daf-4071-b366-596e7ff0f23d"). InnerVolumeSpecName "kube-api-access-xsnkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:24.871637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.871621    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-token" pod "bca2f196-2daf-4071-b366-596e7ff0f23d" (UID: "bca2f196-2daf-4071-b366-596e7ff0f23d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:24.971580 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.971536    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:24.971580 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.971579    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsnkv\" (UniqueName: \"kubernetes.io/projected/bca2f196-2daf-4071-b366-596e7ff0f23d-kube-api-access-xsnkv\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:24.971799 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.971589    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:24.971799 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:24.971636    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bca2f196-2daf-4071-b366-596e7ff0f23d-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:25.399268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.399234    5078 generic.go:356] "Generic (PLEG): container finished" podID="bca2f196-2daf-4071-b366-596e7ff0f23d" containerID="b79fe0f55cf57f69637487be5c0dd6d37ed5762dc3ea8a751ec0276f48ef2df2" exitCode=0
Oct 30 23:42:25.399840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.399305    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"
Oct 30 23:42:25.399840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.399322    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg" event={"ID":"bca2f196-2daf-4071-b366-596e7ff0f23d","Type":"ContainerDied","Data":"b79fe0f55cf57f69637487be5c0dd6d37ed5762dc3ea8a751ec0276f48ef2df2"}
Oct 30 23:42:25.399840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.399381    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg" event={"ID":"bca2f196-2daf-4071-b366-596e7ff0f23d","Type":"ContainerDied","Data":"9717568152c15da37403a3d2319c2694819727d04334c632d28828d46fe1fd20"}
Oct 30 23:42:25.399840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.399412    5078 scope.go:122] "RemoveContainer" containerID="b79fe0f55cf57f69637487be5c0dd6d37ed5762dc3ea8a751ec0276f48ef2df2"
Oct 30 23:42:25.412511 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.412494    5078 scope.go:122] "RemoveContainer" containerID="866067e7d180f1fefe39467b336b5e9a398c807897dd7dd24fd4c0fe6d92182a"
Oct 30 23:42:25.422250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.422232    5078 scope.go:122] "RemoveContainer" containerID="b79fe0f55cf57f69637487be5c0dd6d37ed5762dc3ea8a751ec0276f48ef2df2"
Oct 30 23:42:25.422544 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:42:25.422521    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b79fe0f55cf57f69637487be5c0dd6d37ed5762dc3ea8a751ec0276f48ef2df2\": container with ID starting with b79fe0f55cf57f69637487be5c0dd6d37ed5762dc3ea8a751ec0276f48ef2df2 not found: ID does not exist" containerID="b79fe0f55cf57f69637487be5c0dd6d37ed5762dc3ea8a751ec0276f48ef2df2"
Oct 30 23:42:25.422659 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.422557    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b79fe0f55cf57f69637487be5c0dd6d37ed5762dc3ea8a751ec0276f48ef2df2"} err="failed to get container status \"b79fe0f55cf57f69637487be5c0dd6d37ed5762dc3ea8a751ec0276f48ef2df2\": rpc error: code = NotFound desc = could not find container \"b79fe0f55cf57f69637487be5c0dd6d37ed5762dc3ea8a751ec0276f48ef2df2\": container with ID starting with b79fe0f55cf57f69637487be5c0dd6d37ed5762dc3ea8a751ec0276f48ef2df2 not found: ID does not exist"
Oct 30 23:42:25.422659 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.422583    5078 scope.go:122] "RemoveContainer" containerID="866067e7d180f1fefe39467b336b5e9a398c807897dd7dd24fd4c0fe6d92182a"
Oct 30 23:42:25.422914 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:42:25.422888    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"866067e7d180f1fefe39467b336b5e9a398c807897dd7dd24fd4c0fe6d92182a\": container with ID starting with 866067e7d180f1fefe39467b336b5e9a398c807897dd7dd24fd4c0fe6d92182a not found: ID does not exist" containerID="866067e7d180f1fefe39467b336b5e9a398c807897dd7dd24fd4c0fe6d92182a"
Oct 30 23:42:25.423010 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.422920    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"866067e7d180f1fefe39467b336b5e9a398c807897dd7dd24fd4c0fe6d92182a"} err="failed to get container status \"866067e7d180f1fefe39467b336b5e9a398c807897dd7dd24fd4c0fe6d92182a\": rpc error: code = NotFound desc = could not find container \"866067e7d180f1fefe39467b336b5e9a398c807897dd7dd24fd4c0fe6d92182a\": container with ID starting with 866067e7d180f1fefe39467b336b5e9a398c807897dd7dd24fd4c0fe6d92182a not found: ID does not exist"
Oct 30 23:42:25.537340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.537305    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg"
Oct 30 23:42:25.569862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.569837    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"]
Oct 30 23:42:25.609804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.609776    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b526ac-openshift-default-f8bcb6b49-bbpvg"]
Oct 30 23:42:25.687960 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.687892    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29c46b1e-954c-4c1d-a709-6044a85b855a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c46b1e-954c-4c1d-a709-6044a85b855a-maas-api-xks-ca\") pod \"29c46b1e-954c-4c1d-a709-6044a85b855a\" (UID: \"29c46b1e-954c-4c1d-a709-6044a85b855a\") "
Oct 30 23:42:25.687960 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.687953    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c46b1e-954c-4c1d-a709-6044a85b855a-kube-api-access-hrfkf\" (UniqueName: \"kubernetes.io/projected/29c46b1e-954c-4c1d-a709-6044a85b855a-kube-api-access-hrfkf\") pod \"29c46b1e-954c-4c1d-a709-6044a85b855a\" (UID: \"29c46b1e-954c-4c1d-a709-6044a85b855a\") "
Oct 30 23:42:25.688183 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.688011    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29c46b1e-954c-4c1d-a709-6044a85b855a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c46b1e-954c-4c1d-a709-6044a85b855a-maas-api-openshift-ca\") pod \"29c46b1e-954c-4c1d-a709-6044a85b855a\" (UID: \"29c46b1e-954c-4c1d-a709-6044a85b855a\") "
Oct 30 23:42:25.688448 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.688422    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c46b1e-954c-4c1d-a709-6044a85b855a-maas-api-openshift-ca" pod "29c46b1e-954c-4c1d-a709-6044a85b855a" (UID: "29c46b1e-954c-4c1d-a709-6044a85b855a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:42:25.688750 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.688694    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c46b1e-954c-4c1d-a709-6044a85b855a-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:25.690148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.690121    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c46b1e-954c-4c1d-a709-6044a85b855a-maas-api-xks-ca" pod "29c46b1e-954c-4c1d-a709-6044a85b855a" (UID: "29c46b1e-954c-4c1d-a709-6044a85b855a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:42:25.690296 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.690275    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c46b1e-954c-4c1d-a709-6044a85b855a-kube-api-access-hrfkf" pod "29c46b1e-954c-4c1d-a709-6044a85b855a" (UID: "29c46b1e-954c-4c1d-a709-6044a85b855a"). InnerVolumeSpecName "kube-api-access-hrfkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:25.790186 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.790151    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c46b1e-954c-4c1d-a709-6044a85b855a-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:25.790186 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:25.790182    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrfkf\" (UniqueName: \"kubernetes.io/projected/29c46b1e-954c-4c1d-a709-6044a85b855a-kube-api-access-hrfkf\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:26.299497 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:26.299465    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bca2f196-2daf-4071-b366-596e7ff0f23d" path="/var/lib/kubelet/pods/bca2f196-2daf-4071-b366-596e7ff0f23d/volumes"
Oct 30 23:42:26.299976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:26.299961    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0f4988d-873c-469e-80b6-652ab2cadb2b" path="/var/lib/kubelet/pods/e0f4988d-873c-469e-80b6-652ab2cadb2b/volumes"
Oct 30 23:42:26.409812 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:26.409786    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg"
Oct 30 23:42:26.410194 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:26.409819    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg" event={"ID":"29c46b1e-954c-4c1d-a709-6044a85b855a","Type":"ContainerDied","Data":"6720e7d5b192759f7cb55bd8ef2117438c567d487bd27cc68be277098748ab93"}
Oct 30 23:42:26.410194 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:26.409867    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6720e7d5b192759f7cb55bd8ef2117438c567d487bd27cc68be277098748ab93"
Oct 30 23:42:31.199622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.199042    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp"]
Oct 30 23:42:31.216311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.212336    5078 cpu_manager.go:414] "containerMap: removing container" podUID="29c46b1e-954c-4c1d-a709-6044a85b855a" containerName="revoke-keys"
Oct 30 23:42:31.216311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.212365    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c46b1e-954c-4c1d-a709-6044a85b855a" containerName="revoke-keys"
Oct 30 23:42:31.216311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.212389    5078 cpu_manager.go:414] "containerMap: removing container" podUID="e0f4988d-873c-469e-80b6-652ab2cadb2b" containerName="istio-proxy"
Oct 30 23:42:31.216311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.212399    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0f4988d-873c-469e-80b6-652ab2cadb2b" containerName="istio-proxy"
Oct 30 23:42:31.216311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.212416    5078 cpu_manager.go:414] "containerMap: removing container" podUID="bca2f196-2daf-4071-b366-596e7ff0f23d" containerName="istio-proxy"
Oct 30 23:42:31.216311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.212424    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bca2f196-2daf-4071-b366-596e7ff0f23d" containerName="istio-proxy"
Oct 30 23:42:31.216311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.212442    5078 cpu_manager.go:414] "containerMap: removing container" podUID="bca2f196-2daf-4071-b366-596e7ff0f23d" containerName="istio-proxy"
Oct 30 23:42:31.216311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.212449    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bca2f196-2daf-4071-b366-596e7ff0f23d" containerName="istio-proxy"
Oct 30 23:42:31.216311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.212653    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c46b1e-954c-4c1d-a709-6044a85b855a" containerName="revoke-keys"
Oct 30 23:42:31.216311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.212669    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0f4988d-873c-469e-80b6-652ab2cadb2b" containerName="istio-proxy"
Oct 30 23:42:31.216311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.212680    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="bca2f196-2daf-4071-b366-596e7ff0f23d" containerName="istio-proxy"
Oct 30 23:42:31.216311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.212692    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="bca2f196-2daf-4071-b366-596e7ff0f23d" containerName="istio-proxy"
Oct 30 23:42:31.219816 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.219794    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp"
Oct 30 23:42:31.298347 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.298317    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp"]
Oct 30 23:42:31.306192 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.305438    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:42:31.306192 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.305543    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:42:31.306192 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.305590    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:42:31.331137 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.331105    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:42:31.400319 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.400280    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c4f5f2-82ce-47a5-bfcf-a31feb269947-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp\" (UID: \"63c4f5f2-82ce-47a5-bfcf-a31feb269947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp"
Oct 30 23:42:31.400519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.400334    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hf8m\" (UniqueName: \"kubernetes.io/projected/63c4f5f2-82ce-47a5-bfcf-a31feb269947-kube-api-access-4hf8m\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp\" (UID: \"63c4f5f2-82ce-47a5-bfcf-a31feb269947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp"
Oct 30 23:42:31.400519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.400442    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c4f5f2-82ce-47a5-bfcf-a31feb269947-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp\" (UID: \"63c4f5f2-82ce-47a5-bfcf-a31feb269947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp"
Oct 30 23:42:31.501475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.501389    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c4f5f2-82ce-47a5-bfcf-a31feb269947-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp\" (UID: \"63c4f5f2-82ce-47a5-bfcf-a31feb269947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp"
Oct 30 23:42:31.501475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.501428    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hf8m\" (UniqueName: \"kubernetes.io/projected/63c4f5f2-82ce-47a5-bfcf-a31feb269947-kube-api-access-4hf8m\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp\" (UID: \"63c4f5f2-82ce-47a5-bfcf-a31feb269947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp"
Oct 30 23:42:31.501680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.501466    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c4f5f2-82ce-47a5-bfcf-a31feb269947-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp\" (UID: \"63c4f5f2-82ce-47a5-bfcf-a31feb269947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp"
Oct 30 23:42:31.502212 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.502194    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c4f5f2-82ce-47a5-bfcf-a31feb269947-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp\" (UID: \"63c4f5f2-82ce-47a5-bfcf-a31feb269947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp"
Oct 30 23:42:31.503947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.503927    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c4f5f2-82ce-47a5-bfcf-a31feb269947-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp\" (UID: \"63c4f5f2-82ce-47a5-bfcf-a31feb269947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp"
Oct 30 23:42:31.558975 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.558942    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hf8m\" (UniqueName: \"kubernetes.io/projected/63c4f5f2-82ce-47a5-bfcf-a31feb269947-kube-api-access-4hf8m\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp\" (UID: \"63c4f5f2-82ce-47a5-bfcf-a31feb269947\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp"
Oct 30 23:42:31.595785 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.595748    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp"
Oct 30 23:42:31.975791 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:31.975760    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp"]
Oct 30 23:42:31.976573 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:42:31.976547    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c4f5f2_82ce_47a5_bfcf_a31feb269947.slice/crio-aa3af55e55e15b419d2e44c8e95db90296c5851bc5d97ba2eaf8ac859a2ac71d WatchSource:0}: Error finding container aa3af55e55e15b419d2e44c8e95db90296c5851bc5d97ba2eaf8ac859a2ac71d: Status 404 returned error can't find the container with id aa3af55e55e15b419d2e44c8e95db90296c5851bc5d97ba2eaf8ac859a2ac71d
Oct 30 23:42:32.528216 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:32.528121    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp" event={"ID":"63c4f5f2-82ce-47a5-bfcf-a31feb269947","Type":"ContainerStarted","Data":"0e5b38f592fadbd3596440a4916de19c1943d903fbd6e934848a10e53eba81a7"}
Oct 30 23:42:32.528829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:32.528803    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp" event={"ID":"63c4f5f2-82ce-47a5-bfcf-a31feb269947","Type":"ContainerStarted","Data":"aa3af55e55e15b419d2e44c8e95db90296c5851bc5d97ba2eaf8ac859a2ac71d"}
Oct 30 23:42:32.646352 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:32.646296    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp" podStartSLOduration=1.646282422 podStartE2EDuration="1.646282422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:32.644148663 +0000 UTC m=+2138.532055018" watchObservedRunningTime="2026-10-30 23:42:32.646282422 +0000 UTC m=+2138.534188776"
Oct 30 23:42:33.413941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:33.413910    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"]
Oct 30 23:42:33.414149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:33.414097    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq" podUID="fd9c44ad-4eb5-47bd-946b-5274d535dfb8" containerName="istio-proxy" containerID="cri-o://1335eb21df951f27911888d6ba67ab358156f07c5df254c33e2add9bfa46b654" gracePeriod=30
Oct 30 23:42:33.717560 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:33.712968    5078 generic.go:356] "Generic (PLEG): container finished" podID="63c4f5f2-82ce-47a5-bfcf-a31feb269947" containerID="0e5b38f592fadbd3596440a4916de19c1943d903fbd6e934848a10e53eba81a7" exitCode=0
Oct 30 23:42:33.717560 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:33.713253    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp" event={"ID":"63c4f5f2-82ce-47a5-bfcf-a31feb269947","Type":"ContainerDied","Data":"0e5b38f592fadbd3596440a4916de19c1943d903fbd6e934848a10e53eba81a7"}
Oct 30 23:42:33.979679 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:33.974513    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw"]
Oct 30 23:42:34.935316 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:34.935291    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp"
Oct 30 23:42:35.032202 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:35.032124    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"]
Oct 30 23:42:35.032351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:35.032326    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf" podUID="d23a8a20-2deb-4b62-ba67-e1ae1d15463c" containerName="istio-proxy" containerID="cri-o://42b714639ffd32286b83a316a61b57cda91bd5fa857d095d9a6dc528cc59722e" gracePeriod=30
Oct 30 23:42:35.040220 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:35.040196    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c4f5f2-82ce-47a5-bfcf-a31feb269947-kube-api-access-4hf8m\" (UniqueName: \"kubernetes.io/projected/63c4f5f2-82ce-47a5-bfcf-a31feb269947-kube-api-access-4hf8m\") pod \"63c4f5f2-82ce-47a5-bfcf-a31feb269947\" (UID: \"63c4f5f2-82ce-47a5-bfcf-a31feb269947\") "
Oct 30 23:42:35.040373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:35.040342    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c4f5f2-82ce-47a5-bfcf-a31feb269947-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c4f5f2-82ce-47a5-bfcf-a31feb269947-maas-api-xks-ca\") pod \"63c4f5f2-82ce-47a5-bfcf-a31feb269947\" (UID: \"63c4f5f2-82ce-47a5-bfcf-a31feb269947\") "
Oct 30 23:42:35.040444 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:35.040428    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c4f5f2-82ce-47a5-bfcf-a31feb269947-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c4f5f2-82ce-47a5-bfcf-a31feb269947-maas-api-openshift-ca\") pod \"63c4f5f2-82ce-47a5-bfcf-a31feb269947\" (UID: \"63c4f5f2-82ce-47a5-bfcf-a31feb269947\") "
Oct 30 23:42:35.040861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:35.040829    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c4f5f2-82ce-47a5-bfcf-a31feb269947-maas-api-openshift-ca" pod "63c4f5f2-82ce-47a5-bfcf-a31feb269947" (UID: "63c4f5f2-82ce-47a5-bfcf-a31feb269947"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:42:35.041131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:35.041110    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c4f5f2-82ce-47a5-bfcf-a31feb269947-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:35.043183 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:35.043152    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c4f5f2-82ce-47a5-bfcf-a31feb269947-kube-api-access-4hf8m" pod "63c4f5f2-82ce-47a5-bfcf-a31feb269947" (UID: "63c4f5f2-82ce-47a5-bfcf-a31feb269947"). InnerVolumeSpecName "kube-api-access-4hf8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:35.044977 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:35.044952    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c4f5f2-82ce-47a5-bfcf-a31feb269947-maas-api-xks-ca" pod "63c4f5f2-82ce-47a5-bfcf-a31feb269947" (UID: "63c4f5f2-82ce-47a5-bfcf-a31feb269947"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:42:35.145126 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:35.145038    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c4f5f2-82ce-47a5-bfcf-a31feb269947-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:35.145418 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:35.145396    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hf8m\" (UniqueName: \"kubernetes.io/projected/63c4f5f2-82ce-47a5-bfcf-a31feb269947-kube-api-access-4hf8m\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:35.932738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:35.906227    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp" event={"ID":"63c4f5f2-82ce-47a5-bfcf-a31feb269947","Type":"ContainerDied","Data":"aa3af55e55e15b419d2e44c8e95db90296c5851bc5d97ba2eaf8ac859a2ac71d"}
Oct 30 23:42:35.932738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:35.906331    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa3af55e55e15b419d2e44c8e95db90296c5851bc5d97ba2eaf8ac859a2ac71d"
Oct 30 23:42:35.932738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:35.906412    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp"
Oct 30 23:42:37.356140 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:37.356102    5078 patch_prober.go:29] interesting pod/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:42:37.356539 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:37.356161    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf" podUID="d23a8a20-2deb-4b62-ba67-e1ae1d15463c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:42:38.727709 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.727677    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:42:38.852624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.851878    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-workload-socket\") pod \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") "
Oct 30 23:42:38.852624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.852002    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-data\") pod \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") "
Oct 30 23:42:38.852624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.852080    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-podinfo\") pod \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") "
Oct 30 23:42:38.852624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.852177    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-credential-socket\") pod \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") "
Oct 30 23:42:38.852624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.852226    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-envoy\") pod \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") "
Oct 30 23:42:38.852624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.852253    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istiod-ca-cert\") pod \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") "
Oct 30 23:42:38.852624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.852361    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-kube-api-access-2fqh8\" (UniqueName: \"kubernetes.io/projected/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-kube-api-access-2fqh8\") pod \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") "
Oct 30 23:42:38.852624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.852436    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-token\" (UniqueName: \"kubernetes.io/projected/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-token\") pod \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") "
Oct 30 23:42:38.852624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.852521    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-workload-certs\") pod \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\" (UID: \"fd9c44ad-4eb5-47bd-946b-5274d535dfb8\") "
Oct 30 23:42:38.853165 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.853089    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-workload-certs" pod "fd9c44ad-4eb5-47bd-946b-5274d535dfb8" (UID: "fd9c44ad-4eb5-47bd-946b-5274d535dfb8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:38.866628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.853858    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:38.866821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.866740    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-credential-socket" pod "fd9c44ad-4eb5-47bd-946b-5274d535dfb8" (UID: "fd9c44ad-4eb5-47bd-946b-5274d535dfb8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:38.903661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.876331    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-workload-socket" pod "fd9c44ad-4eb5-47bd-946b-5274d535dfb8" (UID: "fd9c44ad-4eb5-47bd-946b-5274d535dfb8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:38.903661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.876743    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istiod-ca-cert" pod "fd9c44ad-4eb5-47bd-946b-5274d535dfb8" (UID: "fd9c44ad-4eb5-47bd-946b-5274d535dfb8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:42:38.903661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.879764    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-data" pod "fd9c44ad-4eb5-47bd-946b-5274d535dfb8" (UID: "fd9c44ad-4eb5-47bd-946b-5274d535dfb8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:38.975840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.959808    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:38.975840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.959844    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:38.975840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.959859    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:38.975840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.959875    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:38.975840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:38.975420    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw" podUID="72d8a560-396f-486c-b602-5496dfa56ada" containerName="payload-pre-processing" containerID="cri-o://5fc787b400adbae5cc4ab6fb2dfc46647b00e9fddc72ae57ffb5fdbfa2e7ae0f" gracePeriod=25
Oct 30 23:42:39.047502 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.047454    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-token" pod "fd9c44ad-4eb5-47bd-946b-5274d535dfb8" (UID: "fd9c44ad-4eb5-47bd-946b-5274d535dfb8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:39.048771 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.048538    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-envoy" pod "fd9c44ad-4eb5-47bd-946b-5274d535dfb8" (UID: "fd9c44ad-4eb5-47bd-946b-5274d535dfb8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:39.048771 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.048650    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-podinfo" pod "fd9c44ad-4eb5-47bd-946b-5274d535dfb8" (UID: "fd9c44ad-4eb5-47bd-946b-5274d535dfb8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:42:39.048771 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.048738    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-kube-api-access-2fqh8" pod "fd9c44ad-4eb5-47bd-946b-5274d535dfb8" (UID: "fd9c44ad-4eb5-47bd-946b-5274d535dfb8"). InnerVolumeSpecName "kube-api-access-2fqh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:39.058146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.054957    5078 generic.go:356] "Generic (PLEG): container finished" podID="fd9c44ad-4eb5-47bd-946b-5274d535dfb8" containerID="1335eb21df951f27911888d6ba67ab358156f07c5df254c33e2add9bfa46b654" exitCode=0
Oct 30 23:42:39.058146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.055078    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq" event={"ID":"fd9c44ad-4eb5-47bd-946b-5274d535dfb8","Type":"ContainerDied","Data":"1335eb21df951f27911888d6ba67ab358156f07c5df254c33e2add9bfa46b654"}
Oct 30 23:42:39.058146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.055180    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq" event={"ID":"fd9c44ad-4eb5-47bd-946b-5274d535dfb8","Type":"ContainerDied","Data":"abf9bdc2a4a588ac91051dff8ef5540af8f6064e5b6211be80449669514866d2"}
Oct 30 23:42:39.058146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.055245    5078 scope.go:122] "RemoveContainer" containerID="1335eb21df951f27911888d6ba67ab358156f07c5df254c33e2add9bfa46b654"
Oct 30 23:42:39.058146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.055689    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"
Oct 30 23:42:39.061556 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.061468    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:39.061556 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.061498    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:39.061556 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.061515    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fqh8\" (UniqueName: \"kubernetes.io/projected/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-kube-api-access-2fqh8\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:39.061556 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.061534    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd9c44ad-4eb5-47bd-946b-5274d535dfb8-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:39.131805 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.131779    5078 scope.go:122] "RemoveContainer" containerID="1335eb21df951f27911888d6ba67ab358156f07c5df254c33e2add9bfa46b654"
Oct 30 23:42:39.132833 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:42:39.132756    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1335eb21df951f27911888d6ba67ab358156f07c5df254c33e2add9bfa46b654\": container with ID starting with 1335eb21df951f27911888d6ba67ab358156f07c5df254c33e2add9bfa46b654 not found: ID does not exist" containerID="1335eb21df951f27911888d6ba67ab358156f07c5df254c33e2add9bfa46b654"
Oct 30 23:42:39.132833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.132793    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1335eb21df951f27911888d6ba67ab358156f07c5df254c33e2add9bfa46b654"} err="failed to get container status \"1335eb21df951f27911888d6ba67ab358156f07c5df254c33e2add9bfa46b654\": rpc error: code = NotFound desc = could not find container \"1335eb21df951f27911888d6ba67ab358156f07c5df254c33e2add9bfa46b654\": container with ID starting with 1335eb21df951f27911888d6ba67ab358156f07c5df254c33e2add9bfa46b654 not found: ID does not exist"
Oct 30 23:42:39.237156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.237067    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"]
Oct 30 23:42:39.243055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.243025    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nbqlq"]
Oct 30 23:42:39.293238 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.293215    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw"
Oct 30 23:42:39.366454 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.366428    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72d8a560-396f-486c-b602-5496dfa56ada-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d8a560-396f-486c-b602-5496dfa56ada-plugins-config-volume\") pod \"72d8a560-396f-486c-b602-5496dfa56ada\" (UID: \"72d8a560-396f-486c-b602-5496dfa56ada\") "
Oct 30 23:42:39.366654 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.366627    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72d8a560-396f-486c-b602-5496dfa56ada-kube-api-access-bcklw\" (UniqueName: \"kubernetes.io/projected/72d8a560-396f-486c-b602-5496dfa56ada-kube-api-access-bcklw\") pod \"72d8a560-396f-486c-b602-5496dfa56ada\" (UID: \"72d8a560-396f-486c-b602-5496dfa56ada\") "
Oct 30 23:42:39.366855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.366832    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72d8a560-396f-486c-b602-5496dfa56ada-plugins-config-volume" pod "72d8a560-396f-486c-b602-5496dfa56ada" (UID: "72d8a560-396f-486c-b602-5496dfa56ada"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:42:39.367221 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.367196    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72d8a560-396f-486c-b602-5496dfa56ada-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:39.368930 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.368910    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72d8a560-396f-486c-b602-5496dfa56ada-kube-api-access-bcklw" pod "72d8a560-396f-486c-b602-5496dfa56ada" (UID: "72d8a560-396f-486c-b602-5496dfa56ada"). InnerVolumeSpecName "kube-api-access-bcklw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:39.468586 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:39.468554    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcklw\" (UniqueName: \"kubernetes.io/projected/72d8a560-396f-486c-b602-5496dfa56ada-kube-api-access-bcklw\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:40.015430 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.015384    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4"]
Oct 30 23:42:40.021800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.021766    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1ce657-847g4"]
Oct 30 23:42:40.066027 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.065985    5078 generic.go:356] "Generic (PLEG): container finished" podID="72d8a560-396f-486c-b602-5496dfa56ada" containerID="5fc787b400adbae5cc4ab6fb2dfc46647b00e9fddc72ae57ffb5fdbfa2e7ae0f" exitCode=0
Oct 30 23:42:40.066174 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.066059    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw"
Oct 30 23:42:40.066174 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.066084    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw" event={"ID":"72d8a560-396f-486c-b602-5496dfa56ada","Type":"ContainerDied","Data":"5fc787b400adbae5cc4ab6fb2dfc46647b00e9fddc72ae57ffb5fdbfa2e7ae0f"}
Oct 30 23:42:40.066174 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.066157    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw" event={"ID":"72d8a560-396f-486c-b602-5496dfa56ada","Type":"ContainerDied","Data":"e4509eb3bcacafbc0c06cad08bd62245705e4a766a40634cc67310b3dd66f9da"}
Oct 30 23:42:40.066334 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.066182    5078 scope.go:122] "RemoveContainer" containerID="5fc787b400adbae5cc4ab6fb2dfc46647b00e9fddc72ae57ffb5fdbfa2e7ae0f"
Oct 30 23:42:40.147089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.147061    5078 scope.go:122] "RemoveContainer" containerID="5fc787b400adbae5cc4ab6fb2dfc46647b00e9fddc72ae57ffb5fdbfa2e7ae0f"
Oct 30 23:42:40.147522 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:42:40.147492    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fc787b400adbae5cc4ab6fb2dfc46647b00e9fddc72ae57ffb5fdbfa2e7ae0f\": container with ID starting with 5fc787b400adbae5cc4ab6fb2dfc46647b00e9fddc72ae57ffb5fdbfa2e7ae0f not found: ID does not exist" containerID="5fc787b400adbae5cc4ab6fb2dfc46647b00e9fddc72ae57ffb5fdbfa2e7ae0f"
Oct 30 23:42:40.147679 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.147535    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fc787b400adbae5cc4ab6fb2dfc46647b00e9fddc72ae57ffb5fdbfa2e7ae0f"} err="failed to get container status \"5fc787b400adbae5cc4ab6fb2dfc46647b00e9fddc72ae57ffb5fdbfa2e7ae0f\": rpc error: code = NotFound desc = could not find container \"5fc787b400adbae5cc4ab6fb2dfc46647b00e9fddc72ae57ffb5fdbfa2e7ae0f\": container with ID starting with 5fc787b400adbae5cc4ab6fb2dfc46647b00e9fddc72ae57ffb5fdbfa2e7ae0f not found: ID does not exist"
Oct 30 23:42:40.251869 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.251834    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw"]
Oct 30 23:42:40.255923 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.255889    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-196ff7-d8957547f-p8grw"]
Oct 30 23:42:40.300883 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.300853    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69ea360a-a327-4d6c-aa7c-ac30541eba92" path="/var/lib/kubelet/pods/69ea360a-a327-4d6c-aa7c-ac30541eba92/volumes"
Oct 30 23:42:40.301489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.301471    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72d8a560-396f-486c-b602-5496dfa56ada" path="/var/lib/kubelet/pods/72d8a560-396f-486c-b602-5496dfa56ada/volumes"
Oct 30 23:42:40.302145 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.302127    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd9c44ad-4eb5-47bd-946b-5274d535dfb8" path="/var/lib/kubelet/pods/fd9c44ad-4eb5-47bd-946b-5274d535dfb8/volumes"
Oct 30 23:42:40.332394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.332368    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:42:40.485316 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.485286    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-token\" (UniqueName: \"kubernetes.io/projected/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-token\") pod \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") "
Oct 30 23:42:40.485526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.485349    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-credential-socket\") pod \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") "
Oct 30 23:42:40.485526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.485418    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-workload-socket\") pod \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") "
Oct 30 23:42:40.485526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.485463    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-podinfo\") pod \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") "
Oct 30 23:42:40.485731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.485537    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istiod-ca-cert\") pod \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") "
Oct 30 23:42:40.485731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.485682    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-kube-api-access-mnwt6\" (UniqueName: \"kubernetes.io/projected/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-kube-api-access-mnwt6\") pod \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") "
Oct 30 23:42:40.485850 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.485785    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-envoy\") pod \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") "
Oct 30 23:42:40.485850 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.485834    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-workload-certs\") pod \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") "
Oct 30 23:42:40.486917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.485966    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-data\") pod \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\" (UID: \"d23a8a20-2deb-4b62-ba67-e1ae1d15463c\") "
Oct 30 23:42:40.486917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.485977    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-credential-socket" pod "d23a8a20-2deb-4b62-ba67-e1ae1d15463c" (UID: "d23a8a20-2deb-4b62-ba67-e1ae1d15463c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:40.486917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.486042    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-workload-socket" pod "d23a8a20-2deb-4b62-ba67-e1ae1d15463c" (UID: "d23a8a20-2deb-4b62-ba67-e1ae1d15463c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:40.486917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.486492    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istiod-ca-cert" pod "d23a8a20-2deb-4b62-ba67-e1ae1d15463c" (UID: "d23a8a20-2deb-4b62-ba67-e1ae1d15463c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:42:40.486917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.486529    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-workload-certs" pod "d23a8a20-2deb-4b62-ba67-e1ae1d15463c" (UID: "d23a8a20-2deb-4b62-ba67-e1ae1d15463c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:40.486917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.486569    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-data" pod "d23a8a20-2deb-4b62-ba67-e1ae1d15463c" (UID: "d23a8a20-2deb-4b62-ba67-e1ae1d15463c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:40.486917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.486829    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:40.486917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.486852    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:40.486917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.486867    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:40.486917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.486879    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:40.486917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.486892    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:40.487933 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.487906    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-token" pod "d23a8a20-2deb-4b62-ba67-e1ae1d15463c" (UID: "d23a8a20-2deb-4b62-ba67-e1ae1d15463c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:40.488501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.488471    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-podinfo" pod "d23a8a20-2deb-4b62-ba67-e1ae1d15463c" (UID: "d23a8a20-2deb-4b62-ba67-e1ae1d15463c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:42:40.489767 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.489727    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-envoy" pod "d23a8a20-2deb-4b62-ba67-e1ae1d15463c" (UID: "d23a8a20-2deb-4b62-ba67-e1ae1d15463c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:40.490323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.490300    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-kube-api-access-mnwt6" pod "d23a8a20-2deb-4b62-ba67-e1ae1d15463c" (UID: "d23a8a20-2deb-4b62-ba67-e1ae1d15463c"). InnerVolumeSpecName "kube-api-access-mnwt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:40.588517 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.588447    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnwt6\" (UniqueName: \"kubernetes.io/projected/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-kube-api-access-mnwt6\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:40.588517 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.588474    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:40.588517 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.588483    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:40.588517 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:40.588492    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d23a8a20-2deb-4b62-ba67-e1ae1d15463c-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:41.074450 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:41.074417    5078 generic.go:356] "Generic (PLEG): container finished" podID="d23a8a20-2deb-4b62-ba67-e1ae1d15463c" containerID="42b714639ffd32286b83a316a61b57cda91bd5fa857d095d9a6dc528cc59722e" exitCode=0
Oct 30 23:42:41.074940 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:41.074466    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"
Oct 30 23:42:41.074940 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:41.074501    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf" event={"ID":"d23a8a20-2deb-4b62-ba67-e1ae1d15463c","Type":"ContainerDied","Data":"42b714639ffd32286b83a316a61b57cda91bd5fa857d095d9a6dc528cc59722e"}
Oct 30 23:42:41.074940 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:41.074555    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf" event={"ID":"d23a8a20-2deb-4b62-ba67-e1ae1d15463c","Type":"ContainerDied","Data":"064882dd81de291286994a1f0a8d3a2b9bbc15ff3751b8d29d61fa421703909b"}
Oct 30 23:42:41.074940 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:41.074578    5078 scope.go:122] "RemoveContainer" containerID="42b714639ffd32286b83a316a61b57cda91bd5fa857d095d9a6dc528cc59722e"
Oct 30 23:42:41.086685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:41.086662    5078 scope.go:122] "RemoveContainer" containerID="42b714639ffd32286b83a316a61b57cda91bd5fa857d095d9a6dc528cc59722e"
Oct 30 23:42:41.086977 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:42:41.086958    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b714639ffd32286b83a316a61b57cda91bd5fa857d095d9a6dc528cc59722e\": container with ID starting with 42b714639ffd32286b83a316a61b57cda91bd5fa857d095d9a6dc528cc59722e not found: ID does not exist" containerID="42b714639ffd32286b83a316a61b57cda91bd5fa857d095d9a6dc528cc59722e"
Oct 30 23:42:41.087036 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:41.086984    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b714639ffd32286b83a316a61b57cda91bd5fa857d095d9a6dc528cc59722e"} err="failed to get container status \"42b714639ffd32286b83a316a61b57cda91bd5fa857d095d9a6dc528cc59722e\": rpc error: code = NotFound desc = could not find container \"42b714639ffd32286b83a316a61b57cda91bd5fa857d095d9a6dc528cc59722e\": container with ID starting with 42b714639ffd32286b83a316a61b57cda91bd5fa857d095d9a6dc528cc59722e not found: ID does not exist"
Oct 30 23:42:41.312418 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:41.312377    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"]
Oct 30 23:42:41.361614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:41.361523    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-k2jrf"]
Oct 30 23:42:42.095722 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:42.095690    5078 patch_prober.go:29] interesting pod/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:42:42.096135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:42.095762    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9" podUID="1ad7971e-1d0f-4404-883a-76bcf6da8b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:42:42.300315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:42.300273    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d23a8a20-2deb-4b62-ba67-e1ae1d15463c" path="/var/lib/kubelet/pods/d23a8a20-2deb-4b62-ba67-e1ae1d15463c/volumes"
Oct 30 23:42:43.101415 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:43.101380    5078 generic.go:356] "Generic (PLEG): container finished" podID="1ad7971e-1d0f-4404-883a-76bcf6da8b48" containerID="380ae4650e11749b1d329d95cffe581517f0a75900bfdb25ddb8feb200711d92" exitCode=0
Oct 30 23:42:43.101885 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:43.101459    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9" event={"ID":"1ad7971e-1d0f-4404-883a-76bcf6da8b48","Type":"ContainerDied","Data":"380ae4650e11749b1d329d95cffe581517f0a75900bfdb25ddb8feb200711d92"}
Oct 30 23:42:43.101992 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:43.101975    5078 scope.go:122] "RemoveContainer" containerID="380ae4650e11749b1d329d95cffe581517f0a75900bfdb25ddb8feb200711d92"
Oct 30 23:42:43.104007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:43.103971    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:42:43.104101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:43.104057    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:42:43.104141 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:43.104098    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:42:44.111000 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:44.110963    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9" event={"ID":"1ad7971e-1d0f-4404-883a-76bcf6da8b48","Type":"ContainerStarted","Data":"c19e4e5ab62393984d7997f3f0f57fe3901376375afc835a7e845529f4eb9044"}
Oct 30 23:42:45.090109 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:45.090075    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:42:45.090322 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:45.090247    5078 patch_prober.go:29] interesting pod/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 23:42:45.090322 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:45.090299    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9" podUID="1ad7971e-1d0f-4404-883a-76bcf6da8b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 23:42:46.089363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:46.089335    5078 patch_prober.go:29] interesting pod/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 23:42:46.089873 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:46.089392    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9" podUID="1ad7971e-1d0f-4404-883a-76bcf6da8b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 23:42:46.693800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:46.693762    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"]
Oct 30 23:42:46.694125 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:46.694090    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz" podUID="0b13c626-bbb0-4874-88b2-cecb77a61423" containerName="istio-proxy" containerID="cri-o://6f6623d2d72a32cfb982b097626d6693f122f62d9ffe0cdf3ca0d458329a12d5" gracePeriod=30
Oct 30 23:42:47.090087 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.090049    5078 patch_prober.go:29] interesting pod/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 23:42:47.090477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.090117    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9" podUID="1ad7971e-1d0f-4404-883a-76bcf6da8b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 23:42:47.554927 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.554899    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv"]
Oct 30 23:42:47.555465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.555452    5078 cpu_manager.go:414] "containerMap: removing container" podUID="63c4f5f2-82ce-47a5-bfcf-a31feb269947" containerName="revoke-keys"
Oct 30 23:42:47.555515 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.555466    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c4f5f2-82ce-47a5-bfcf-a31feb269947" containerName="revoke-keys"
Oct 30 23:42:47.555515 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.555481    5078 cpu_manager.go:414] "containerMap: removing container" podUID="fd9c44ad-4eb5-47bd-946b-5274d535dfb8" containerName="istio-proxy"
Oct 30 23:42:47.555515 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.555486    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd9c44ad-4eb5-47bd-946b-5274d535dfb8" containerName="istio-proxy"
Oct 30 23:42:47.555515 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.555505    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d23a8a20-2deb-4b62-ba67-e1ae1d15463c" containerName="istio-proxy"
Oct 30 23:42:47.555515 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.555510    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d23a8a20-2deb-4b62-ba67-e1ae1d15463c" containerName="istio-proxy"
Oct 30 23:42:47.555705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.555529    5078 cpu_manager.go:414] "containerMap: removing container" podUID="72d8a560-396f-486c-b602-5496dfa56ada" containerName="payload-pre-processing"
Oct 30 23:42:47.555705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.555535    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72d8a560-396f-486c-b602-5496dfa56ada" containerName="payload-pre-processing"
Oct 30 23:42:47.555705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.555652    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="72d8a560-396f-486c-b602-5496dfa56ada" containerName="payload-pre-processing"
Oct 30 23:42:47.555705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.555664    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c4f5f2-82ce-47a5-bfcf-a31feb269947" containerName="revoke-keys"
Oct 30 23:42:47.555705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.555671    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd9c44ad-4eb5-47bd-946b-5274d535dfb8" containerName="istio-proxy"
Oct 30 23:42:47.555705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.555680    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d23a8a20-2deb-4b62-ba67-e1ae1d15463c" containerName="istio-proxy"
Oct 30 23:42:47.578870 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.578841    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv"
Oct 30 23:42:47.616703 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:42:47.616649    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:42:47.616923 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.616895    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv"]
Oct 30 23:42:47.639317 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.639288    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv\" (UID: \"99e66e43-6a0c-4b97-be01-a65a5ec5fd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv"
Oct 30 23:42:47.639481 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.639368    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv\" (UID: \"99e66e43-6a0c-4b97-be01-a65a5ec5fd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv"
Oct 30 23:42:47.639535 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.639514    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zfvk\" (UniqueName: \"kubernetes.io/projected/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-kube-api-access-8zfvk\") pod \"maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv\" (UID: \"99e66e43-6a0c-4b97-be01-a65a5ec5fd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv"
Oct 30 23:42:47.681865 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.681827    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:42:47.681865 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.681837    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:42:47.682130 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.681925    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:42:47.682285 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.682260    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:42:47.740861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.740834    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv\" (UID: \"99e66e43-6a0c-4b97-be01-a65a5ec5fd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv"
Oct 30 23:42:47.741037 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.740893    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv\" (UID: \"99e66e43-6a0c-4b97-be01-a65a5ec5fd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv"
Oct 30 23:42:47.741037 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.740969    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zfvk\" (UniqueName: \"kubernetes.io/projected/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-kube-api-access-8zfvk\") pod \"maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv\" (UID: \"99e66e43-6a0c-4b97-be01-a65a5ec5fd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv"
Oct 30 23:42:47.741650 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.741628    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv\" (UID: \"99e66e43-6a0c-4b97-be01-a65a5ec5fd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv"
Oct 30 23:42:47.743949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.743916    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv\" (UID: \"99e66e43-6a0c-4b97-be01-a65a5ec5fd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv"
Oct 30 23:42:47.798897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.798865    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zfvk\" (UniqueName: \"kubernetes.io/projected/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-kube-api-access-8zfvk\") pod \"maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv\" (UID: \"99e66e43-6a0c-4b97-be01-a65a5ec5fd70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv"
Oct 30 23:42:47.900513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.900429    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv"
Oct 30 23:42:47.937790 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.937762    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h"]
Oct 30 23:42:47.942028 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:47.942010    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h"
Oct 30 23:42:48.001259 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:48.001227    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h"]
Oct 30 23:42:48.044799 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:48.044761    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c1392b-ce18-4918-b612-1f1424dafd7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h\" (UID: \"b9c1392b-ce18-4918-b612-1f1424dafd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h"
Oct 30 23:42:48.045344 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:48.044958    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c1392b-ce18-4918-b612-1f1424dafd7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h\" (UID: \"b9c1392b-ce18-4918-b612-1f1424dafd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h"
Oct 30 23:42:48.045344 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:48.045126    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t9zb\" (UniqueName: \"kubernetes.io/projected/b9c1392b-ce18-4918-b612-1f1424dafd7e-kube-api-access-7t9zb\") pod \"maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h\" (UID: \"b9c1392b-ce18-4918-b612-1f1424dafd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h"
Oct 30 23:42:48.090480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:48.090440    5078 patch_prober.go:29] interesting pod/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 23:42:48.090938 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:48.090500    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9" podUID="1ad7971e-1d0f-4404-883a-76bcf6da8b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 23:42:48.146186 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:48.146156    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c1392b-ce18-4918-b612-1f1424dafd7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h\" (UID: \"b9c1392b-ce18-4918-b612-1f1424dafd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h"
Oct 30 23:42:48.146375 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:48.146235    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t9zb\" (UniqueName: \"kubernetes.io/projected/b9c1392b-ce18-4918-b612-1f1424dafd7e-kube-api-access-7t9zb\") pod \"maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h\" (UID: \"b9c1392b-ce18-4918-b612-1f1424dafd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h"
Oct 30 23:42:48.146375 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:48.146343    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c1392b-ce18-4918-b612-1f1424dafd7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h\" (UID: \"b9c1392b-ce18-4918-b612-1f1424dafd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h"
Oct 30 23:42:48.146990 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:48.146962    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c1392b-ce18-4918-b612-1f1424dafd7e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h\" (UID: \"b9c1392b-ce18-4918-b612-1f1424dafd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h"
Oct 30 23:42:48.148892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:48.148871    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c1392b-ce18-4918-b612-1f1424dafd7e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h\" (UID: \"b9c1392b-ce18-4918-b612-1f1424dafd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h"
Oct 30 23:42:48.204080 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:48.204016    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t9zb\" (UniqueName: \"kubernetes.io/projected/b9c1392b-ce18-4918-b612-1f1424dafd7e-kube-api-access-7t9zb\") pod \"maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h\" (UID: \"b9c1392b-ce18-4918-b612-1f1424dafd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h"
Oct 30 23:42:48.282890 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:48.282552    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h"
Oct 30 23:42:48.305361 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:48.305335    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv"]
Oct 30 23:42:48.307071 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:42:48.307044    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e66e43_6a0c_4b97_be01_a65a5ec5fd70.slice/crio-1306f153ab82d4afc421f66007febceec2877de56095dca42ba96893d2c3864b WatchSource:0}: Error finding container 1306f153ab82d4afc421f66007febceec2877de56095dca42ba96893d2c3864b: Status 404 returned error can't find the container with id 1306f153ab82d4afc421f66007febceec2877de56095dca42ba96893d2c3864b
Oct 30 23:42:48.689739 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:48.689708    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h"]
Oct 30 23:42:48.691500 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:42:48.691472    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c1392b_ce18_4918_b612_1f1424dafd7e.slice/crio-7946148cbb2367762e3e3cb7d3fa5f7ab6319cfec70855d03b53124ff18b4a76 WatchSource:0}: Error finding container 7946148cbb2367762e3e3cb7d3fa5f7ab6319cfec70855d03b53124ff18b4a76: Status 404 returned error can't find the container with id 7946148cbb2367762e3e3cb7d3fa5f7ab6319cfec70855d03b53124ff18b4a76
Oct 30 23:42:49.090585 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:49.090559    5078 patch_prober.go:29] interesting pod/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Oct 30 23:42:49.090956 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:49.090635    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9" podUID="1ad7971e-1d0f-4404-883a-76bcf6da8b48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Oct 30 23:42:49.147562 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:49.147522    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv" event={"ID":"99e66e43-6a0c-4b97-be01-a65a5ec5fd70","Type":"ContainerStarted","Data":"6a48c6925313c77111ab381634a9e51394430d5be13c5151fdad24d99ec71986"}
Oct 30 23:42:49.147709 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:49.147590    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv" event={"ID":"99e66e43-6a0c-4b97-be01-a65a5ec5fd70","Type":"ContainerStarted","Data":"1306f153ab82d4afc421f66007febceec2877de56095dca42ba96893d2c3864b"}
Oct 30 23:42:49.149266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:49.149242    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h" event={"ID":"b9c1392b-ce18-4918-b612-1f1424dafd7e","Type":"ContainerStarted","Data":"0fecb3f2c8898997df58451c03079a25d87383410f7d2129ca96a22fb165f8b7"}
Oct 30 23:42:49.149347 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:49.149284    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h" event={"ID":"b9c1392b-ce18-4918-b612-1f1424dafd7e","Type":"ContainerStarted","Data":"7946148cbb2367762e3e3cb7d3fa5f7ab6319cfec70855d03b53124ff18b4a76"}
Oct 30 23:42:49.283508 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:49.280578    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv" podStartSLOduration=2.280558257 podStartE2EDuration="2.280558257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:49.277153205 +0000 UTC m=+2155.165059559" watchObservedRunningTime="2026-10-30 23:42:49.280558257 +0000 UTC m=+2155.168464610"
Oct 30 23:42:49.390480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:49.390422    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h" podStartSLOduration=2.390403417 podStartE2EDuration="2.390403417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:42:49.389328879 +0000 UTC m=+2155.277235256" watchObservedRunningTime="2026-10-30 23:42:49.390403417 +0000 UTC m=+2155.278309776"
Oct 30 23:42:49.775955 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:42:49.775921    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c1392b_ce18_4918_b612_1f1424dafd7e.slice/crio-0fecb3f2c8898997df58451c03079a25d87383410f7d2129ca96a22fb165f8b7.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 23:42:50.094378 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:50.094350    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:42:50.094912 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:50.094894    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:42:50.095508 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:50.095473    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:42:50.157469 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:50.157441    5078 generic.go:356] "Generic (PLEG): container finished" podID="99e66e43-6a0c-4b97-be01-a65a5ec5fd70" containerID="6a48c6925313c77111ab381634a9e51394430d5be13c5151fdad24d99ec71986" exitCode=0
Oct 30 23:42:50.157590 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:50.157565    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv" event={"ID":"99e66e43-6a0c-4b97-be01-a65a5ec5fd70","Type":"ContainerDied","Data":"6a48c6925313c77111ab381634a9e51394430d5be13c5151fdad24d99ec71986"}
Oct 30 23:42:50.159075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:50.159057    5078 generic.go:356] "Generic (PLEG): container finished" podID="b9c1392b-ce18-4918-b612-1f1424dafd7e" containerID="0fecb3f2c8898997df58451c03079a25d87383410f7d2129ca96a22fb165f8b7" exitCode=0
Oct 30 23:42:50.159179 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:50.159131    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h" event={"ID":"b9c1392b-ce18-4918-b612-1f1424dafd7e","Type":"ContainerDied","Data":"0fecb3f2c8898997df58451c03079a25d87383410f7d2129ca96a22fb165f8b7"}
Oct 30 23:42:51.332187 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.331893    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h"
Oct 30 23:42:51.369289 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.369265    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv"
Oct 30 23:42:51.415285 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.415263    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-kube-api-access-8zfvk\" (UniqueName: \"kubernetes.io/projected/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-kube-api-access-8zfvk\") pod \"99e66e43-6a0c-4b97-be01-a65a5ec5fd70\" (UID: \"99e66e43-6a0c-4b97-be01-a65a5ec5fd70\") "
Oct 30 23:42:51.415423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.415323    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9c1392b-ce18-4918-b612-1f1424dafd7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c1392b-ce18-4918-b612-1f1424dafd7e-maas-api-openshift-ca\") pod \"b9c1392b-ce18-4918-b612-1f1424dafd7e\" (UID: \"b9c1392b-ce18-4918-b612-1f1424dafd7e\") "
Oct 30 23:42:51.415423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.415379    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c1392b-ce18-4918-b612-1f1424dafd7e-kube-api-access-7t9zb\" (UniqueName: \"kubernetes.io/projected/b9c1392b-ce18-4918-b612-1f1424dafd7e-kube-api-access-7t9zb\") pod \"b9c1392b-ce18-4918-b612-1f1424dafd7e\" (UID: \"b9c1392b-ce18-4918-b612-1f1424dafd7e\") "
Oct 30 23:42:51.415676 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.415658    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-maas-api-xks-ca\") pod \"99e66e43-6a0c-4b97-be01-a65a5ec5fd70\" (UID: \"99e66e43-6a0c-4b97-be01-a65a5ec5fd70\") "
Oct 30 23:42:51.415762 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.415685    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c1392b-ce18-4918-b612-1f1424dafd7e-maas-api-openshift-ca" pod "b9c1392b-ce18-4918-b612-1f1424dafd7e" (UID: "b9c1392b-ce18-4918-b612-1f1424dafd7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:42:51.415762 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.415714    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9c1392b-ce18-4918-b612-1f1424dafd7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c1392b-ce18-4918-b612-1f1424dafd7e-maas-api-xks-ca\") pod \"b9c1392b-ce18-4918-b612-1f1424dafd7e\" (UID: \"b9c1392b-ce18-4918-b612-1f1424dafd7e\") "
Oct 30 23:42:51.415861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.415756    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-maas-api-openshift-ca\") pod \"99e66e43-6a0c-4b97-be01-a65a5ec5fd70\" (UID: \"99e66e43-6a0c-4b97-be01-a65a5ec5fd70\") "
Oct 30 23:42:51.416332 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.416280    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c1392b-ce18-4918-b612-1f1424dafd7e-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:51.416418 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.416364    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-maas-api-openshift-ca" pod "99e66e43-6a0c-4b97-be01-a65a5ec5fd70" (UID: "99e66e43-6a0c-4b97-be01-a65a5ec5fd70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:42:51.417807 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.417776    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-kube-api-access-8zfvk" pod "99e66e43-6a0c-4b97-be01-a65a5ec5fd70" (UID: "99e66e43-6a0c-4b97-be01-a65a5ec5fd70"). InnerVolumeSpecName "kube-api-access-8zfvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:51.418052 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.418031    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-maas-api-xks-ca" pod "99e66e43-6a0c-4b97-be01-a65a5ec5fd70" (UID: "99e66e43-6a0c-4b97-be01-a65a5ec5fd70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:42:51.418263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.418245    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c1392b-ce18-4918-b612-1f1424dafd7e-kube-api-access-7t9zb" pod "b9c1392b-ce18-4918-b612-1f1424dafd7e" (UID: "b9c1392b-ce18-4918-b612-1f1424dafd7e"). InnerVolumeSpecName "kube-api-access-7t9zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:51.418315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.418257    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c1392b-ce18-4918-b612-1f1424dafd7e-maas-api-xks-ca" pod "b9c1392b-ce18-4918-b612-1f1424dafd7e" (UID: "b9c1392b-ce18-4918-b612-1f1424dafd7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:42:51.517506 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.517475    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:51.517506 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.517503    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c1392b-ce18-4918-b612-1f1424dafd7e-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:51.517506 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.517513    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:51.517733 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.517522    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zfvk\" (UniqueName: \"kubernetes.io/projected/99e66e43-6a0c-4b97-be01-a65a5ec5fd70-kube-api-access-8zfvk\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:51.517733 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.517532    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t9zb\" (UniqueName: \"kubernetes.io/projected/b9c1392b-ce18-4918-b612-1f1424dafd7e-kube-api-access-7t9zb\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:51.946569 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:51.946538    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:42:52.024736 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.024659    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-token\" (UniqueName: \"kubernetes.io/projected/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-token\") pod \"0b13c626-bbb0-4874-88b2-cecb77a61423\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") "
Oct 30 23:42:52.024736 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.024709    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-workload-certs\") pod \"0b13c626-bbb0-4874-88b2-cecb77a61423\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") "
Oct 30 23:42:52.024916 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.024776    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b13c626-bbb0-4874-88b2-cecb77a61423-kube-api-access-bl7nt\" (UniqueName: \"kubernetes.io/projected/0b13c626-bbb0-4874-88b2-cecb77a61423-kube-api-access-bl7nt\") pod \"0b13c626-bbb0-4874-88b2-cecb77a61423\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") "
Oct 30 23:42:52.024916 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.024841    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-data\") pod \"0b13c626-bbb0-4874-88b2-cecb77a61423\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") "
Oct 30 23:42:52.025016 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.024945    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-workload-socket\") pod \"0b13c626-bbb0-4874-88b2-cecb77a61423\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") "
Oct 30 23:42:52.025016 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.024965    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b13c626-bbb0-4874-88b2-cecb77a61423-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b13c626-bbb0-4874-88b2-cecb77a61423-istiod-ca-cert\") pod \"0b13c626-bbb0-4874-88b2-cecb77a61423\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") "
Oct 30 23:42:52.025125 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.025042    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-envoy\") pod \"0b13c626-bbb0-4874-88b2-cecb77a61423\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") "
Oct 30 23:42:52.025125 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.025077    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-podinfo\") pod \"0b13c626-bbb0-4874-88b2-cecb77a61423\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") "
Oct 30 23:42:52.025125 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.025077    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-workload-certs" pod "0b13c626-bbb0-4874-88b2-cecb77a61423" (UID: "0b13c626-bbb0-4874-88b2-cecb77a61423"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:52.025288 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.025127    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-credential-socket\") pod \"0b13c626-bbb0-4874-88b2-cecb77a61423\" (UID: \"0b13c626-bbb0-4874-88b2-cecb77a61423\") "
Oct 30 23:42:52.025288 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.025195    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-data" pod "0b13c626-bbb0-4874-88b2-cecb77a61423" (UID: "0b13c626-bbb0-4874-88b2-cecb77a61423"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:52.025516 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.025458    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-credential-socket" pod "0b13c626-bbb0-4874-88b2-cecb77a61423" (UID: "0b13c626-bbb0-4874-88b2-cecb77a61423"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:52.025516 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.025485    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b13c626-bbb0-4874-88b2-cecb77a61423-istiod-ca-cert" pod "0b13c626-bbb0-4874-88b2-cecb77a61423" (UID: "0b13c626-bbb0-4874-88b2-cecb77a61423"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:42:52.025718 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.025622    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-workload-socket" pod "0b13c626-bbb0-4874-88b2-cecb77a61423" (UID: "0b13c626-bbb0-4874-88b2-cecb77a61423"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:52.025810 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.025794    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:52.025869 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.025814    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b13c626-bbb0-4874-88b2-cecb77a61423-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:52.025869 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.025823    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:52.025869 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.025834    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:52.025869 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.025847    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:52.027786 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.027762    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-podinfo" pod "0b13c626-bbb0-4874-88b2-cecb77a61423" (UID: "0b13c626-bbb0-4874-88b2-cecb77a61423"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:42:52.027898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.027823    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-envoy" pod "0b13c626-bbb0-4874-88b2-cecb77a61423" (UID: "0b13c626-bbb0-4874-88b2-cecb77a61423"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:42:52.027898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.027823    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-token" pod "0b13c626-bbb0-4874-88b2-cecb77a61423" (UID: "0b13c626-bbb0-4874-88b2-cecb77a61423"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:52.027898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.027838    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b13c626-bbb0-4874-88b2-cecb77a61423-kube-api-access-bl7nt" pod "0b13c626-bbb0-4874-88b2-cecb77a61423" (UID: "0b13c626-bbb0-4874-88b2-cecb77a61423"). InnerVolumeSpecName "kube-api-access-bl7nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:52.126964 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.126934    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:52.126964 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.126965    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:52.127163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.126975    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b13c626-bbb0-4874-88b2-cecb77a61423-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:52.127163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.126983    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl7nt\" (UniqueName: \"kubernetes.io/projected/0b13c626-bbb0-4874-88b2-cecb77a61423-kube-api-access-bl7nt\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:52.174584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.174557    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv"
Oct 30 23:42:52.174584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.174610    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv" event={"ID":"99e66e43-6a0c-4b97-be01-a65a5ec5fd70","Type":"ContainerDied","Data":"1306f153ab82d4afc421f66007febceec2877de56095dca42ba96893d2c3864b"}
Oct 30 23:42:52.174848 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.174657    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1306f153ab82d4afc421f66007febceec2877de56095dca42ba96893d2c3864b"
Oct 30 23:42:52.176176 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.176155    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h"
Oct 30 23:42:52.176304 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.176200    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h" event={"ID":"b9c1392b-ce18-4918-b612-1f1424dafd7e","Type":"ContainerDied","Data":"7946148cbb2367762e3e3cb7d3fa5f7ab6319cfec70855d03b53124ff18b4a76"}
Oct 30 23:42:52.176304 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.176244    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7946148cbb2367762e3e3cb7d3fa5f7ab6319cfec70855d03b53124ff18b4a76"
Oct 30 23:42:52.178270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.178244    5078 generic.go:356] "Generic (PLEG): container finished" podID="0b13c626-bbb0-4874-88b2-cecb77a61423" containerID="6f6623d2d72a32cfb982b097626d6693f122f62d9ffe0cdf3ca0d458329a12d5" exitCode=0
Oct 30 23:42:52.178414 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.178295    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"
Oct 30 23:42:52.178414 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.178307    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz" event={"ID":"0b13c626-bbb0-4874-88b2-cecb77a61423","Type":"ContainerDied","Data":"6f6623d2d72a32cfb982b097626d6693f122f62d9ffe0cdf3ca0d458329a12d5"}
Oct 30 23:42:52.178414 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.178343    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz" event={"ID":"0b13c626-bbb0-4874-88b2-cecb77a61423","Type":"ContainerDied","Data":"1e1d8dac0652ce2017820fc7d7d427604ec42ca06d01d130584c67365a35d8ba"}
Oct 30 23:42:52.178414 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.178365    5078 scope.go:122] "RemoveContainer" containerID="6f6623d2d72a32cfb982b097626d6693f122f62d9ffe0cdf3ca0d458329a12d5"
Oct 30 23:42:52.192304 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.192268    5078 scope.go:122] "RemoveContainer" containerID="e813ebaa8a7b9cdc591c908ab710b4afc85a904f813c38e3c7d98a99eb3079b9"
Oct 30 23:42:52.203384 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.203366    5078 scope.go:122] "RemoveContainer" containerID="6f6623d2d72a32cfb982b097626d6693f122f62d9ffe0cdf3ca0d458329a12d5"
Oct 30 23:42:52.203684 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:42:52.203665    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f6623d2d72a32cfb982b097626d6693f122f62d9ffe0cdf3ca0d458329a12d5\": container with ID starting with 6f6623d2d72a32cfb982b097626d6693f122f62d9ffe0cdf3ca0d458329a12d5 not found: ID does not exist" containerID="6f6623d2d72a32cfb982b097626d6693f122f62d9ffe0cdf3ca0d458329a12d5"
Oct 30 23:42:52.203738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.203691    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f6623d2d72a32cfb982b097626d6693f122f62d9ffe0cdf3ca0d458329a12d5"} err="failed to get container status \"6f6623d2d72a32cfb982b097626d6693f122f62d9ffe0cdf3ca0d458329a12d5\": rpc error: code = NotFound desc = could not find container \"6f6623d2d72a32cfb982b097626d6693f122f62d9ffe0cdf3ca0d458329a12d5\": container with ID starting with 6f6623d2d72a32cfb982b097626d6693f122f62d9ffe0cdf3ca0d458329a12d5 not found: ID does not exist"
Oct 30 23:42:52.203738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.203709    5078 scope.go:122] "RemoveContainer" containerID="e813ebaa8a7b9cdc591c908ab710b4afc85a904f813c38e3c7d98a99eb3079b9"
Oct 30 23:42:52.203931 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:42:52.203914    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e813ebaa8a7b9cdc591c908ab710b4afc85a904f813c38e3c7d98a99eb3079b9\": container with ID starting with e813ebaa8a7b9cdc591c908ab710b4afc85a904f813c38e3c7d98a99eb3079b9 not found: ID does not exist" containerID="e813ebaa8a7b9cdc591c908ab710b4afc85a904f813c38e3c7d98a99eb3079b9"
Oct 30 23:42:52.203976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.203937    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e813ebaa8a7b9cdc591c908ab710b4afc85a904f813c38e3c7d98a99eb3079b9"} err="failed to get container status \"e813ebaa8a7b9cdc591c908ab710b4afc85a904f813c38e3c7d98a99eb3079b9\": rpc error: code = NotFound desc = could not find container \"e813ebaa8a7b9cdc591c908ab710b4afc85a904f813c38e3c7d98a99eb3079b9\": container with ID starting with e813ebaa8a7b9cdc591c908ab710b4afc85a904f813c38e3c7d98a99eb3079b9 not found: ID does not exist"
Oct 30 23:42:52.458655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.458627    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"]
Oct 30 23:42:52.461972 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:52.461948    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1ce657-openshift-default-54fc6db5bc-h4gfz"]
Oct 30 23:42:53.320957 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.320931    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8"]
Oct 30 23:42:53.321522 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.321507    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0b13c626-bbb0-4874-88b2-cecb77a61423" containerName="istio-proxy"
Oct 30 23:42:53.321570 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.321524    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b13c626-bbb0-4874-88b2-cecb77a61423" containerName="istio-proxy"
Oct 30 23:42:53.321570 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.321540    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0b13c626-bbb0-4874-88b2-cecb77a61423" containerName="istio-proxy"
Oct 30 23:42:53.321570 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.321545    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b13c626-bbb0-4874-88b2-cecb77a61423" containerName="istio-proxy"
Oct 30 23:42:53.321686 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.321574    5078 cpu_manager.go:414] "containerMap: removing container" podUID="99e66e43-6a0c-4b97-be01-a65a5ec5fd70" containerName="revoke-keys"
Oct 30 23:42:53.321686 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.321579    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99e66e43-6a0c-4b97-be01-a65a5ec5fd70" containerName="revoke-keys"
Oct 30 23:42:53.321686 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.321615    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b9c1392b-ce18-4918-b612-1f1424dafd7e" containerName="revoke-keys"
Oct 30 23:42:53.321686 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.321620    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c1392b-ce18-4918-b612-1f1424dafd7e" containerName="revoke-keys"
Oct 30 23:42:53.321808 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.321708    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="99e66e43-6a0c-4b97-be01-a65a5ec5fd70" containerName="revoke-keys"
Oct 30 23:42:53.321808 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.321719    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c1392b-ce18-4918-b612-1f1424dafd7e" containerName="revoke-keys"
Oct 30 23:42:53.321808 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.321725    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b13c626-bbb0-4874-88b2-cecb77a61423" containerName="istio-proxy"
Oct 30 23:42:53.321808 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.321732    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b13c626-bbb0-4874-88b2-cecb77a61423" containerName="istio-proxy"
Oct 30 23:42:53.326568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.326548    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8"
Oct 30 23:42:53.380658 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.380631    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8"]
Oct 30 23:42:53.426379 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.426356    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:42:53.426540 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.426473    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:42:53.426540 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.426525    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:42:53.428493 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.428463    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:42:53.442872 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.442847    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8\" (UID: \"f46e2510-bd1d-41c2-9ca0-e785f5bb0133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8"
Oct 30 23:42:53.443075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.443059    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8\" (UID: \"f46e2510-bd1d-41c2-9ca0-e785f5bb0133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8"
Oct 30 23:42:53.443162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.443143    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rdzt\" (UniqueName: \"kubernetes.io/projected/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-kube-api-access-8rdzt\") pod \"maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8\" (UID: \"f46e2510-bd1d-41c2-9ca0-e785f5bb0133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8"
Oct 30 23:42:53.544422 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.544395    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8\" (UID: \"f46e2510-bd1d-41c2-9ca0-e785f5bb0133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8"
Oct 30 23:42:53.544909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.544439    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rdzt\" (UniqueName: \"kubernetes.io/projected/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-kube-api-access-8rdzt\") pod \"maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8\" (UID: \"f46e2510-bd1d-41c2-9ca0-e785f5bb0133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8"
Oct 30 23:42:53.544909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.544639    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8\" (UID: \"f46e2510-bd1d-41c2-9ca0-e785f5bb0133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8"
Oct 30 23:42:53.545238 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.545217    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8\" (UID: \"f46e2510-bd1d-41c2-9ca0-e785f5bb0133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8"
Oct 30 23:42:53.546972 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.546952    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8\" (UID: \"f46e2510-bd1d-41c2-9ca0-e785f5bb0133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8"
Oct 30 23:42:53.603133 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.603064    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rdzt\" (UniqueName: \"kubernetes.io/projected/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-kube-api-access-8rdzt\") pod \"maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8\" (UID: \"f46e2510-bd1d-41c2-9ca0-e785f5bb0133\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8"
Oct 30 23:42:53.640411 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:53.640384    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8"
Oct 30 23:42:54.015978 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:54.015950    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8"]
Oct 30 23:42:54.016722 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:42:54.016696    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46e2510_bd1d_41c2_9ca0_e785f5bb0133.slice/crio-1c7e4a78fedc616ccdf6bd4d452b9d73fa3fec78a52268710f54e254ebc38ec1 WatchSource:0}: Error finding container 1c7e4a78fedc616ccdf6bd4d452b9d73fa3fec78a52268710f54e254ebc38ec1: Status 404 returned error can't find the container with id 1c7e4a78fedc616ccdf6bd4d452b9d73fa3fec78a52268710f54e254ebc38ec1
Oct 30 23:42:54.194739 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:54.194710    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8" event={"ID":"f46e2510-bd1d-41c2-9ca0-e785f5bb0133","Type":"ContainerStarted","Data":"1c7e4a78fedc616ccdf6bd4d452b9d73fa3fec78a52268710f54e254ebc38ec1"}
Oct 30 23:42:54.302578 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:54.302548    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b13c626-bbb0-4874-88b2-cecb77a61423" path="/var/lib/kubelet/pods/0b13c626-bbb0-4874-88b2-cecb77a61423/volumes"
Oct 30 23:42:55.203339 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:55.203253    5078 generic.go:356] "Generic (PLEG): container finished" podID="f46e2510-bd1d-41c2-9ca0-e785f5bb0133" containerID="b1f694f9aae6dc0d25121618008122dfca8c9b8a447fd07ad61c23de3c798aae" exitCode=0
Oct 30 23:42:55.203733 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:55.203325    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8" event={"ID":"f46e2510-bd1d-41c2-9ca0-e785f5bb0133","Type":"ContainerDied","Data":"b1f694f9aae6dc0d25121618008122dfca8c9b8a447fd07ad61c23de3c798aae"}
Oct 30 23:42:56.289807 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:56.288509    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl"]
Oct 30 23:42:56.399985 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:56.399954    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt"]
Oct 30 23:42:56.406810 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:56.406789    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8"
Oct 30 23:42:56.507936 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:56.500252    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-kube-api-access-8rdzt\" (UniqueName: \"kubernetes.io/projected/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-kube-api-access-8rdzt\") pod \"f46e2510-bd1d-41c2-9ca0-e785f5bb0133\" (UID: \"f46e2510-bd1d-41c2-9ca0-e785f5bb0133\") "
Oct 30 23:42:56.507936 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:56.500394    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-maas-api-xks-ca\") pod \"f46e2510-bd1d-41c2-9ca0-e785f5bb0133\" (UID: \"f46e2510-bd1d-41c2-9ca0-e785f5bb0133\") "
Oct 30 23:42:56.507936 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:56.500534    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-maas-api-openshift-ca\") pod \"f46e2510-bd1d-41c2-9ca0-e785f5bb0133\" (UID: \"f46e2510-bd1d-41c2-9ca0-e785f5bb0133\") "
Oct 30 23:42:56.507936 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:56.501066    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-maas-api-openshift-ca" pod "f46e2510-bd1d-41c2-9ca0-e785f5bb0133" (UID: "f46e2510-bd1d-41c2-9ca0-e785f5bb0133"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:42:56.507936 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:56.505963    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:56.507936 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:56.506570    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-kube-api-access-8rdzt" pod "f46e2510-bd1d-41c2-9ca0-e785f5bb0133" (UID: "f46e2510-bd1d-41c2-9ca0-e785f5bb0133"). InnerVolumeSpecName "kube-api-access-8rdzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:42:56.509519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:56.509494    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-maas-api-xks-ca" pod "f46e2510-bd1d-41c2-9ca0-e785f5bb0133" (UID: "f46e2510-bd1d-41c2-9ca0-e785f5bb0133"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:42:56.608092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:56.608057    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:56.608092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:56.608093    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rdzt\" (UniqueName: \"kubernetes.io/projected/f46e2510-bd1d-41c2-9ca0-e785f5bb0133-kube-api-access-8rdzt\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:42:57.219962 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:57.219924    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8"
Oct 30 23:42:57.219962 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:57.219950    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8" event={"ID":"f46e2510-bd1d-41c2-9ca0-e785f5bb0133","Type":"ContainerDied","Data":"1c7e4a78fedc616ccdf6bd4d452b9d73fa3fec78a52268710f54e254ebc38ec1"}
Oct 30 23:42:57.220194 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:57.219996    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c7e4a78fedc616ccdf6bd4d452b9d73fa3fec78a52268710f54e254ebc38ec1"
Oct 30 23:42:57.384355 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:57.384321    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"]
Oct 30 23:42:57.385175 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:42:57.385144    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz" podUID="7a62b6f7-1528-4540-8793-c333094fb427" containerName="istio-proxy" containerID="cri-o://9254b9beda66889b5d4685f847cc27e6d808560898e3f85a639bd8dc27f5561f" gracePeriod=30
Oct 30 23:43:01.289957 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:01.289863    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl" podUID="260ce73f-2f49-43e7-a62f-9bcd51c63075" containerName="payload-processing" containerID="cri-o://7dec1821bf7011f55a564b2b10d247755f777390014c9ccc7412b2ab5471ae89" gracePeriod=25
Oct 30 23:43:01.400703 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:01.400644    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt" podUID="8395dcd7-d923-41f9-984e-ecf53b137eb2" containerName="payload-pre-processing" containerID="cri-o://137ccbbb3d53b718db77fe631f0844a33ebfbe02c959a78a29b1b82d214242bf" gracePeriod=25
Oct 30 23:43:01.413618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:01.413570    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:43:01.413707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:01.413651    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz" podUID="7a62b6f7-1528-4540-8793-c333094fb427" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:43:01.578071 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:01.578048    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl"
Oct 30 23:43:01.626995 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:01.626968    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260ce73f-2f49-43e7-a62f-9bcd51c63075-kube-api-access-xj7b8\" (UniqueName: \"kubernetes.io/projected/260ce73f-2f49-43e7-a62f-9bcd51c63075-kube-api-access-xj7b8\") pod \"260ce73f-2f49-43e7-a62f-9bcd51c63075\" (UID: \"260ce73f-2f49-43e7-a62f-9bcd51c63075\") "
Oct 30 23:43:01.627190 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:01.627127    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260ce73f-2f49-43e7-a62f-9bcd51c63075-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260ce73f-2f49-43e7-a62f-9bcd51c63075-plugins-config-volume\") pod \"260ce73f-2f49-43e7-a62f-9bcd51c63075\" (UID: \"260ce73f-2f49-43e7-a62f-9bcd51c63075\") "
Oct 30 23:43:01.628282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:01.628248    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260ce73f-2f49-43e7-a62f-9bcd51c63075-plugins-config-volume" pod "260ce73f-2f49-43e7-a62f-9bcd51c63075" (UID: "260ce73f-2f49-43e7-a62f-9bcd51c63075"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:01.630755 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:01.630733    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260ce73f-2f49-43e7-a62f-9bcd51c63075-kube-api-access-xj7b8" pod "260ce73f-2f49-43e7-a62f-9bcd51c63075" (UID: "260ce73f-2f49-43e7-a62f-9bcd51c63075"). InnerVolumeSpecName "kube-api-access-xj7b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:01.685676 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:01.685648    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt"
Oct 30 23:43:01.728931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:01.728902    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8395dcd7-d923-41f9-984e-ecf53b137eb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8395dcd7-d923-41f9-984e-ecf53b137eb2-plugins-config-volume\") pod \"8395dcd7-d923-41f9-984e-ecf53b137eb2\" (UID: \"8395dcd7-d923-41f9-984e-ecf53b137eb2\") "
Oct 30 23:43:01.729102 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:01.728954    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8395dcd7-d923-41f9-984e-ecf53b137eb2-kube-api-access-x527s\" (UniqueName: \"kubernetes.io/projected/8395dcd7-d923-41f9-984e-ecf53b137eb2-kube-api-access-x527s\") pod \"8395dcd7-d923-41f9-984e-ecf53b137eb2\" (UID: \"8395dcd7-d923-41f9-984e-ecf53b137eb2\") "
Oct 30 23:43:01.729308 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:01.729283    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8395dcd7-d923-41f9-984e-ecf53b137eb2-plugins-config-volume" pod "8395dcd7-d923-41f9-984e-ecf53b137eb2" (UID: "8395dcd7-d923-41f9-984e-ecf53b137eb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:01.729924 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:01.729899    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260ce73f-2f49-43e7-a62f-9bcd51c63075-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:01.730033 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:01.729929    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8395dcd7-d923-41f9-984e-ecf53b137eb2-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:01.730033 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:01.729945    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj7b8\" (UniqueName: \"kubernetes.io/projected/260ce73f-2f49-43e7-a62f-9bcd51c63075-kube-api-access-xj7b8\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:01.731295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:01.731268    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8395dcd7-d923-41f9-984e-ecf53b137eb2-kube-api-access-x527s" pod "8395dcd7-d923-41f9-984e-ecf53b137eb2" (UID: "8395dcd7-d923-41f9-984e-ecf53b137eb2"). InnerVolumeSpecName "kube-api-access-x527s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:01.831692 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:01.831590    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x527s\" (UniqueName: \"kubernetes.io/projected/8395dcd7-d923-41f9-984e-ecf53b137eb2-kube-api-access-x527s\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.334416 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.334383    5078 generic.go:356] "Generic (PLEG): container finished" podID="260ce73f-2f49-43e7-a62f-9bcd51c63075" containerID="7dec1821bf7011f55a564b2b10d247755f777390014c9ccc7412b2ab5471ae89" exitCode=0
Oct 30 23:43:02.334843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.334497    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl"
Oct 30 23:43:02.334843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.334517    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl" event={"ID":"260ce73f-2f49-43e7-a62f-9bcd51c63075","Type":"ContainerDied","Data":"7dec1821bf7011f55a564b2b10d247755f777390014c9ccc7412b2ab5471ae89"}
Oct 30 23:43:02.334843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.334612    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl" event={"ID":"260ce73f-2f49-43e7-a62f-9bcd51c63075","Type":"ContainerDied","Data":"322c4712ec6561cdaff6746aaeb785a120b19b1eb08fc121ed224cfd4fb94804"}
Oct 30 23:43:02.334843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.334639    5078 scope.go:122] "RemoveContainer" containerID="7dec1821bf7011f55a564b2b10d247755f777390014c9ccc7412b2ab5471ae89"
Oct 30 23:43:02.336117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.336091    5078 generic.go:356] "Generic (PLEG): container finished" podID="8395dcd7-d923-41f9-984e-ecf53b137eb2" containerID="137ccbbb3d53b718db77fe631f0844a33ebfbe02c959a78a29b1b82d214242bf" exitCode=0
Oct 30 23:43:02.336231 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.336157    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt"
Oct 30 23:43:02.336231 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.336182    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt" event={"ID":"8395dcd7-d923-41f9-984e-ecf53b137eb2","Type":"ContainerDied","Data":"137ccbbb3d53b718db77fe631f0844a33ebfbe02c959a78a29b1b82d214242bf"}
Oct 30 23:43:02.336352 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.336242    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt" event={"ID":"8395dcd7-d923-41f9-984e-ecf53b137eb2","Type":"ContainerDied","Data":"ed43c1476438827b7ad6a6a7031c64e0646cf928654998818552cc303852f77e"}
Oct 30 23:43:02.348209 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.348189    5078 scope.go:122] "RemoveContainer" containerID="7dec1821bf7011f55a564b2b10d247755f777390014c9ccc7412b2ab5471ae89"
Oct 30 23:43:02.348557 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:43:02.348539    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dec1821bf7011f55a564b2b10d247755f777390014c9ccc7412b2ab5471ae89\": container with ID starting with 7dec1821bf7011f55a564b2b10d247755f777390014c9ccc7412b2ab5471ae89 not found: ID does not exist" containerID="7dec1821bf7011f55a564b2b10d247755f777390014c9ccc7412b2ab5471ae89"
Oct 30 23:43:02.348671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.348568    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dec1821bf7011f55a564b2b10d247755f777390014c9ccc7412b2ab5471ae89"} err="failed to get container status \"7dec1821bf7011f55a564b2b10d247755f777390014c9ccc7412b2ab5471ae89\": rpc error: code = NotFound desc = could not find container \"7dec1821bf7011f55a564b2b10d247755f777390014c9ccc7412b2ab5471ae89\": container with ID starting with 7dec1821bf7011f55a564b2b10d247755f777390014c9ccc7412b2ab5471ae89 not found: ID does not exist"
Oct 30 23:43:02.348671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.348609    5078 scope.go:122] "RemoveContainer" containerID="137ccbbb3d53b718db77fe631f0844a33ebfbe02c959a78a29b1b82d214242bf"
Oct 30 23:43:02.359332 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.359309    5078 scope.go:122] "RemoveContainer" containerID="137ccbbb3d53b718db77fe631f0844a33ebfbe02c959a78a29b1b82d214242bf"
Oct 30 23:43:02.359657 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:43:02.359637    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"137ccbbb3d53b718db77fe631f0844a33ebfbe02c959a78a29b1b82d214242bf\": container with ID starting with 137ccbbb3d53b718db77fe631f0844a33ebfbe02c959a78a29b1b82d214242bf not found: ID does not exist" containerID="137ccbbb3d53b718db77fe631f0844a33ebfbe02c959a78a29b1b82d214242bf"
Oct 30 23:43:02.359741 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.359665    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"137ccbbb3d53b718db77fe631f0844a33ebfbe02c959a78a29b1b82d214242bf"} err="failed to get container status \"137ccbbb3d53b718db77fe631f0844a33ebfbe02c959a78a29b1b82d214242bf\": rpc error: code = NotFound desc = could not find container \"137ccbbb3d53b718db77fe631f0844a33ebfbe02c959a78a29b1b82d214242bf\": container with ID starting with 137ccbbb3d53b718db77fe631f0844a33ebfbe02c959a78a29b1b82d214242bf not found: ID does not exist"
Oct 30 23:43:02.499671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.499638    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl"]
Oct 30 23:43:02.549248 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.549215    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7d1536-797545749-76cgl"]
Oct 30 23:43:02.661889 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.661856    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt"]
Oct 30 23:43:02.665323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.665296    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7d1536-55fb7dc447-ndztt"]
Oct 30 23:43:02.687832 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.687800    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:43:02.744351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.744322    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a62b6f7-1528-4540-8793-c333094fb427-kube-api-access-gc8pt\" (UniqueName: \"kubernetes.io/projected/7a62b6f7-1528-4540-8793-c333094fb427-kube-api-access-gc8pt\") pod \"7a62b6f7-1528-4540-8793-c333094fb427\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") "
Oct 30 23:43:02.744536 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.744360    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-istio-envoy\") pod \"7a62b6f7-1528-4540-8793-c333094fb427\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") "
Oct 30 23:43:02.744536 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.744394    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a62b6f7-1528-4540-8793-c333094fb427-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a62b6f7-1528-4540-8793-c333094fb427-istio-podinfo\") pod \"7a62b6f7-1528-4540-8793-c333094fb427\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") "
Oct 30 23:43:02.744536 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.744414    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-istio-data\") pod \"7a62b6f7-1528-4540-8793-c333094fb427\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") "
Oct 30 23:43:02.744536 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.744466    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a62b6f7-1528-4540-8793-c333094fb427-istio-token\" (UniqueName: \"kubernetes.io/projected/7a62b6f7-1528-4540-8793-c333094fb427-istio-token\") pod \"7a62b6f7-1528-4540-8793-c333094fb427\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") "
Oct 30 23:43:02.744800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.744574    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a62b6f7-1528-4540-8793-c333094fb427-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a62b6f7-1528-4540-8793-c333094fb427-istiod-ca-cert\") pod \"7a62b6f7-1528-4540-8793-c333094fb427\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") "
Oct 30 23:43:02.744800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.744690    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-workload-socket\") pod \"7a62b6f7-1528-4540-8793-c333094fb427\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") "
Oct 30 23:43:02.744908 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.744817    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-workload-certs\") pod \"7a62b6f7-1528-4540-8793-c333094fb427\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") "
Oct 30 23:43:02.744908 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.744828    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-istio-data" pod "7a62b6f7-1528-4540-8793-c333094fb427" (UID: "7a62b6f7-1528-4540-8793-c333094fb427"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:43:02.744908 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.744857    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-credential-socket\") pod \"7a62b6f7-1528-4540-8793-c333094fb427\" (UID: \"7a62b6f7-1528-4540-8793-c333094fb427\") "
Oct 30 23:43:02.745069 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.745010    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-workload-socket" pod "7a62b6f7-1528-4540-8793-c333094fb427" (UID: "7a62b6f7-1528-4540-8793-c333094fb427"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:43:02.745836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.745148    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-workload-certs" pod "7a62b6f7-1528-4540-8793-c333094fb427" (UID: "7a62b6f7-1528-4540-8793-c333094fb427"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:43:02.745836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.745261    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-credential-socket" pod "7a62b6f7-1528-4540-8793-c333094fb427" (UID: "7a62b6f7-1528-4540-8793-c333094fb427"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:43:02.745836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.745332    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a62b6f7-1528-4540-8793-c333094fb427-istiod-ca-cert" pod "7a62b6f7-1528-4540-8793-c333094fb427" (UID: "7a62b6f7-1528-4540-8793-c333094fb427"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:02.745836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.745769    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.745836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.745785    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a62b6f7-1528-4540-8793-c333094fb427-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.745836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.745797    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.745836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.745810    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.745836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.745819    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.747349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.747319    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-istio-envoy" pod "7a62b6f7-1528-4540-8793-c333094fb427" (UID: "7a62b6f7-1528-4540-8793-c333094fb427"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:43:02.747470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.747434    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a62b6f7-1528-4540-8793-c333094fb427-istio-token" pod "7a62b6f7-1528-4540-8793-c333094fb427" (UID: "7a62b6f7-1528-4540-8793-c333094fb427"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:02.747516 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.747469    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a62b6f7-1528-4540-8793-c333094fb427-kube-api-access-gc8pt" pod "7a62b6f7-1528-4540-8793-c333094fb427" (UID: "7a62b6f7-1528-4540-8793-c333094fb427"). InnerVolumeSpecName "kube-api-access-gc8pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:02.747575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.747557    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a62b6f7-1528-4540-8793-c333094fb427-istio-podinfo" pod "7a62b6f7-1528-4540-8793-c333094fb427" (UID: "7a62b6f7-1528-4540-8793-c333094fb427"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:43:02.847454 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.847364    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc8pt\" (UniqueName: \"kubernetes.io/projected/7a62b6f7-1528-4540-8793-c333094fb427-kube-api-access-gc8pt\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.847454 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.847404    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a62b6f7-1528-4540-8793-c333094fb427-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.847454 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.847419    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a62b6f7-1528-4540-8793-c333094fb427-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.848096 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.847433    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a62b6f7-1528-4540-8793-c333094fb427-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:02.902403 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:02.902375    5078 scope.go:122] "RemoveContainer" containerID="9254b9beda66889b5d4685f847cc27e6d808560898e3f85a639bd8dc27f5561f"
Oct 30 23:43:03.343372 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:03.343325    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz" event={"ID":"7a62b6f7-1528-4540-8793-c333094fb427","Type":"ContainerDied","Data":"9254b9beda66889b5d4685f847cc27e6d808560898e3f85a639bd8dc27f5561f"}
Oct 30 23:43:03.343836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:03.343379    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz" event={"ID":"7a62b6f7-1528-4540-8793-c333094fb427","Type":"ContainerDied","Data":"a1bae3f1e1e2fbbc63c461e11744d932f35fb447485362bf19e95e7cd099c270"}
Oct 30 23:43:03.343836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:03.343428    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"
Oct 30 23:43:03.575382 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:03.575349    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"]
Oct 30 23:43:03.580543 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:03.580517    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-mj8mz"]
Oct 30 23:43:04.302850 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:04.302817    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="260ce73f-2f49-43e7-a62f-9bcd51c63075" path="/var/lib/kubelet/pods/260ce73f-2f49-43e7-a62f-9bcd51c63075/volumes"
Oct 30 23:43:04.303306 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:04.303289    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a62b6f7-1528-4540-8793-c333094fb427" path="/var/lib/kubelet/pods/7a62b6f7-1528-4540-8793-c333094fb427/volumes"
Oct 30 23:43:04.303763 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:04.303742    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8395dcd7-d923-41f9-984e-ecf53b137eb2" path="/var/lib/kubelet/pods/8395dcd7-d923-41f9-984e-ecf53b137eb2/volumes"
Oct 30 23:43:05.028846 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:05.028761    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s"]
Oct 30 23:43:05.133713 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:05.133685    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp"]
Oct 30 23:43:05.380534 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:05.380381    5078 generic.go:356] "Generic (PLEG): container finished" podID="4539c4b3-4808-4706-b794-82d7a2fa6034" containerID="370b58a9d4d9006ab5ca5ea920af855449df7322bc79d00cea53277ca522310c" exitCode=0
Oct 30 23:43:05.380768 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:05.380621    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26" event={"ID":"4539c4b3-4808-4706-b794-82d7a2fa6034","Type":"ContainerDied","Data":"370b58a9d4d9006ab5ca5ea920af855449df7322bc79d00cea53277ca522310c"}
Oct 30 23:43:05.381104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:05.381079    5078 scope.go:122] "RemoveContainer" containerID="370b58a9d4d9006ab5ca5ea920af855449df7322bc79d00cea53277ca522310c"
Oct 30 23:43:05.383051 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:05.383020    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:43:05.383132 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:05.383106    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:43:05.383173 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:05.383150    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:43:06.390160 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:06.390125    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26" event={"ID":"4539c4b3-4808-4706-b794-82d7a2fa6034","Type":"ContainerStarted","Data":"669052311b2e8ecc352f4cd6f919b78b81b3eb8773b5b1ea17d77a8150844dac"}
Oct 30 23:43:06.981790 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:06.981746    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:43:06.983483 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:06.983448    5078 patch_prober.go:29] interesting pod/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 23:43:06.983694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:06.983503    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26" podUID="4539c4b3-4808-4706-b794-82d7a2fa6034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 23:43:07.981995 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:07.981953    5078 patch_prober.go:29] interesting pod/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 23:43:07.982500 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:07.982018    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26" podUID="4539c4b3-4808-4706-b794-82d7a2fa6034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 23:43:08.982185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:08.982144    5078 patch_prober.go:29] interesting pod/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 23:43:08.982672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:08.982204    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26" podUID="4539c4b3-4808-4706-b794-82d7a2fa6034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 23:43:09.983093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:09.983054    5078 patch_prober.go:29] interesting pod/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 23:43:09.983502 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:09.983143    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26" podUID="4539c4b3-4808-4706-b794-82d7a2fa6034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 23:43:10.030081 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.030038    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s" podUID="dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9" containerName="payload-processing" containerID="cri-o://705daa61dd248037ce82a1c729786e1c6711057ee49128d473a1e938c066cb9c" gracePeriod=25
Oct 30 23:43:10.134852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.134790    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp" podUID="63f35153-159d-4249-86fe-fcab10ce168d" containerName="payload-pre-processing" containerID="cri-o://b7a258dce8fe640cd77c6884f745ad7c820e7492cd476065b452a63c11d6da47" gracePeriod=25
Oct 30 23:43:10.431886 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.431555    5078 generic.go:356] "Generic (PLEG): container finished" podID="63f35153-159d-4249-86fe-fcab10ce168d" containerID="b7a258dce8fe640cd77c6884f745ad7c820e7492cd476065b452a63c11d6da47" exitCode=0
Oct 30 23:43:10.431886 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.431679    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp" event={"ID":"63f35153-159d-4249-86fe-fcab10ce168d","Type":"ContainerDied","Data":"b7a258dce8fe640cd77c6884f745ad7c820e7492cd476065b452a63c11d6da47"}
Oct 30 23:43:10.435283 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.435204    5078 generic.go:356] "Generic (PLEG): container finished" podID="dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9" containerID="705daa61dd248037ce82a1c729786e1c6711057ee49128d473a1e938c066cb9c" exitCode=0
Oct 30 23:43:10.435283 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.435307    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s" event={"ID":"dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9","Type":"ContainerDied","Data":"705daa61dd248037ce82a1c729786e1c6711057ee49128d473a1e938c066cb9c"}
Oct 30 23:43:10.435583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.435352    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s" event={"ID":"dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9","Type":"ContainerDied","Data":"c73bbdd7f67b579abf043f927c2bb2928852a921364227d858292bd37f9b7536"}
Oct 30 23:43:10.435583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.435375    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c73bbdd7f67b579abf043f927c2bb2928852a921364227d858292bd37f9b7536"
Oct 30 23:43:10.449363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.449338    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s"
Oct 30 23:43:10.509537 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.509516    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp"
Oct 30 23:43:10.631680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.631651    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f35153-159d-4249-86fe-fcab10ce168d-kube-api-access-9mnm9\" (UniqueName: \"kubernetes.io/projected/63f35153-159d-4249-86fe-fcab10ce168d-kube-api-access-9mnm9\") pod \"63f35153-159d-4249-86fe-fcab10ce168d\" (UID: \"63f35153-159d-4249-86fe-fcab10ce168d\") "
Oct 30 23:43:10.631861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.631722    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9-plugins-config-volume\") pod \"dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9\" (UID: \"dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9\") "
Oct 30 23:43:10.631861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.631830    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9-kube-api-access-ppb8z\" (UniqueName: \"kubernetes.io/projected/dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9-kube-api-access-ppb8z\") pod \"dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9\" (UID: \"dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9\") "
Oct 30 23:43:10.631938 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.631875    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f35153-159d-4249-86fe-fcab10ce168d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f35153-159d-4249-86fe-fcab10ce168d-plugins-config-volume\") pod \"63f35153-159d-4249-86fe-fcab10ce168d\" (UID: \"63f35153-159d-4249-86fe-fcab10ce168d\") "
Oct 30 23:43:10.632043 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.632026    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9-plugins-config-volume" pod "dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9" (UID: "dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:10.632257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.632233    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f35153-159d-4249-86fe-fcab10ce168d-plugins-config-volume" pod "63f35153-159d-4249-86fe-fcab10ce168d" (UID: "63f35153-159d-4249-86fe-fcab10ce168d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:10.632493 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.632478    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:10.632544 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.632496    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63f35153-159d-4249-86fe-fcab10ce168d-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:10.634005 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.633982    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f35153-159d-4249-86fe-fcab10ce168d-kube-api-access-9mnm9" pod "63f35153-159d-4249-86fe-fcab10ce168d" (UID: "63f35153-159d-4249-86fe-fcab10ce168d"). InnerVolumeSpecName "kube-api-access-9mnm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:10.634109 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.634028    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9-kube-api-access-ppb8z" pod "dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9" (UID: "dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9"). InnerVolumeSpecName "kube-api-access-ppb8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:10.734295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.734262    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mnm9\" (UniqueName: \"kubernetes.io/projected/63f35153-159d-4249-86fe-fcab10ce168d-kube-api-access-9mnm9\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:10.734295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.734292    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppb8z\" (UniqueName: \"kubernetes.io/projected/dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9-kube-api-access-ppb8z\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:10.982093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.982002    5078 patch_prober.go:29] interesting pod/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 23:43:10.982093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:10.982069    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26" podUID="4539c4b3-4808-4706-b794-82d7a2fa6034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 23:43:11.444234 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:11.444203    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp"
Oct 30 23:43:11.444234 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:11.444225    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp" event={"ID":"63f35153-159d-4249-86fe-fcab10ce168d","Type":"ContainerDied","Data":"16ab6a9ee04a7d194b5e0eb41b2061bbb8adfbfe3f5e4187f07c950372e8fe08"}
Oct 30 23:43:11.444773 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:11.444297    5078 scope.go:122] "RemoveContainer" containerID="b7a258dce8fe640cd77c6884f745ad7c820e7492cd476065b452a63c11d6da47"
Oct 30 23:43:11.444773 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:11.444465    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s"
Oct 30 23:43:11.450235 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:11.450213    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:43:11.609866 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:11.609828    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s"]
Oct 30 23:43:11.614703 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:11.614679    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0a7530-6b8985d9d7-l2b6s"]
Oct 30 23:43:11.784140 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:11.784044    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp"]
Oct 30 23:43:11.787873 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:11.787852    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0a7530-799ff7d555-mq5cp"]
Oct 30 23:43:11.981944 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:11.981910    5078 patch_prober.go:29] interesting pod/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused" start-of-body=
Oct 30 23:43:11.982149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:11.981959    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26" podUID="4539c4b3-4808-4706-b794-82d7a2fa6034" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.173:15021/healthz/ready\": dial tcp 10.133.0.173:15021: connect: connection refused"
Oct 30 23:43:12.299353 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:12.299321    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f35153-159d-4249-86fe-fcab10ce168d" path="/var/lib/kubelet/pods/63f35153-159d-4249-86fe-fcab10ce168d/volumes"
Oct 30 23:43:12.299848 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:12.299834    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9" path="/var/lib/kubelet/pods/dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9/volumes"
Oct 30 23:43:12.986900 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:12.986869    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:43:12.988100 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:12.988076    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:43:16.108933 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.108902    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz"]
Oct 30 23:43:16.109746 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.109727    5078 cpu_manager.go:414] "containerMap: removing container" podUID="63f35153-159d-4249-86fe-fcab10ce168d" containerName="payload-pre-processing"
Oct 30 23:43:16.109813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.109749    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f35153-159d-4249-86fe-fcab10ce168d" containerName="payload-pre-processing"
Oct 30 23:43:16.109813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.109788    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f46e2510-bd1d-41c2-9ca0-e785f5bb0133" containerName="revoke-keys"
Oct 30 23:43:16.109813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.109795    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46e2510-bd1d-41c2-9ca0-e785f5bb0133" containerName="revoke-keys"
Oct 30 23:43:16.109813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.109813    5078 cpu_manager.go:414] "containerMap: removing container" podUID="dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9" containerName="payload-processing"
Oct 30 23:43:16.109939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.109818    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9" containerName="payload-processing"
Oct 30 23:43:16.109939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.109836    5078 cpu_manager.go:414] "containerMap: removing container" podUID="260ce73f-2f49-43e7-a62f-9bcd51c63075" containerName="payload-processing"
Oct 30 23:43:16.109939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.109840    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260ce73f-2f49-43e7-a62f-9bcd51c63075" containerName="payload-processing"
Oct 30 23:43:16.109939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.109854    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7a62b6f7-1528-4540-8793-c333094fb427" containerName="istio-proxy"
Oct 30 23:43:16.109939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.109859    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a62b6f7-1528-4540-8793-c333094fb427" containerName="istio-proxy"
Oct 30 23:43:16.109939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.109872    5078 cpu_manager.go:414] "containerMap: removing container" podUID="8395dcd7-d923-41f9-984e-ecf53b137eb2" containerName="payload-pre-processing"
Oct 30 23:43:16.109939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.109877    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8395dcd7-d923-41f9-984e-ecf53b137eb2" containerName="payload-pre-processing"
Oct 30 23:43:16.110146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.109970    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46e2510-bd1d-41c2-9ca0-e785f5bb0133" containerName="revoke-keys"
Oct 30 23:43:16.110146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.109982    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc94c3c7-8f6b-41aa-9d6f-42dbb71783e9" containerName="payload-processing"
Oct 30 23:43:16.110146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.109990    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="260ce73f-2f49-43e7-a62f-9bcd51c63075" containerName="payload-processing"
Oct 30 23:43:16.110146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.109996    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a62b6f7-1528-4540-8793-c333094fb427" containerName="istio-proxy"
Oct 30 23:43:16.110146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.110002    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="8395dcd7-d923-41f9-984e-ecf53b137eb2" containerName="payload-pre-processing"
Oct 30 23:43:16.110146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.110009    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f35153-159d-4249-86fe-fcab10ce168d" containerName="payload-pre-processing"
Oct 30 23:43:16.114823 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.114805    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz"
Oct 30 23:43:16.177904 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.176868    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:43:16.189620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.180166    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz"]
Oct 30 23:43:16.258998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.217474    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:43:16.258998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.217614    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:43:16.258998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.219985    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djdp7\" (UniqueName: \"kubernetes.io/projected/09f50622-c9d6-43c2-bde6-219e11207233-kube-api-access-djdp7\") pod \"maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz\" (UID: \"09f50622-c9d6-43c2-bde6-219e11207233\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz"
Oct 30 23:43:16.258998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.220041    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f50622-c9d6-43c2-bde6-219e11207233-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz\" (UID: \"09f50622-c9d6-43c2-bde6-219e11207233\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz"
Oct 30 23:43:16.258998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.220096    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f50622-c9d6-43c2-bde6-219e11207233-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz\" (UID: \"09f50622-c9d6-43c2-bde6-219e11207233\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz"
Oct 30 23:43:16.259629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.259581    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:43:16.328185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.328151    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djdp7\" (UniqueName: \"kubernetes.io/projected/09f50622-c9d6-43c2-bde6-219e11207233-kube-api-access-djdp7\") pod \"maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz\" (UID: \"09f50622-c9d6-43c2-bde6-219e11207233\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz"
Oct 30 23:43:16.328463 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.328440    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f50622-c9d6-43c2-bde6-219e11207233-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz\" (UID: \"09f50622-c9d6-43c2-bde6-219e11207233\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz"
Oct 30 23:43:16.328662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.328642    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f50622-c9d6-43c2-bde6-219e11207233-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz\" (UID: \"09f50622-c9d6-43c2-bde6-219e11207233\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz"
Oct 30 23:43:16.329417 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.329384    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f50622-c9d6-43c2-bde6-219e11207233-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz\" (UID: \"09f50622-c9d6-43c2-bde6-219e11207233\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz"
Oct 30 23:43:16.332226 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.332202    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f50622-c9d6-43c2-bde6-219e11207233-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz\" (UID: \"09f50622-c9d6-43c2-bde6-219e11207233\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz"
Oct 30 23:43:16.385134 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.385070    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djdp7\" (UniqueName: \"kubernetes.io/projected/09f50622-c9d6-43c2-bde6-219e11207233-kube-api-access-djdp7\") pod \"maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz\" (UID: \"09f50622-c9d6-43c2-bde6-219e11207233\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz"
Oct 30 23:43:16.429928 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.429892    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz"
Oct 30 23:43:16.804425 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:16.804396    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz"]
Oct 30 23:43:16.805509 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:43:16.805484    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09f50622_c9d6_43c2_bde6_219e11207233.slice/crio-0dbcac3def57d8b7776a7a2bfa5f298aaa46a39dfe8fbf319ae66e5db5401ab4 WatchSource:0}: Error finding container 0dbcac3def57d8b7776a7a2bfa5f298aaa46a39dfe8fbf319ae66e5db5401ab4: Status 404 returned error can't find the container with id 0dbcac3def57d8b7776a7a2bfa5f298aaa46a39dfe8fbf319ae66e5db5401ab4
Oct 30 23:43:17.494559 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:17.494470    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz" event={"ID":"09f50622-c9d6-43c2-bde6-219e11207233","Type":"ContainerStarted","Data":"93b85680f764963017b24c6d10472f57733b70e268462c3d0967520b0c35ce05"}
Oct 30 23:43:17.494559 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:17.494523    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz" event={"ID":"09f50622-c9d6-43c2-bde6-219e11207233","Type":"ContainerStarted","Data":"0dbcac3def57d8b7776a7a2bfa5f298aaa46a39dfe8fbf319ae66e5db5401ab4"}
Oct 30 23:43:17.611650 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:17.610779    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz" podStartSLOduration=1.610765056 podStartE2EDuration="1.610765056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:17.607619678 +0000 UTC m=+2183.495526028" watchObservedRunningTime="2026-10-30 23:43:17.610765056 +0000 UTC m=+2183.498671408"
Oct 30 23:43:17.618193 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:43:17.618158    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:43:18.502691 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:18.502663    5078 generic.go:356] "Generic (PLEG): container finished" podID="09f50622-c9d6-43c2-bde6-219e11207233" containerID="93b85680f764963017b24c6d10472f57733b70e268462c3d0967520b0c35ce05" exitCode=0
Oct 30 23:43:18.503101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:18.502746    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz" event={"ID":"09f50622-c9d6-43c2-bde6-219e11207233","Type":"ContainerDied","Data":"93b85680f764963017b24c6d10472f57733b70e268462c3d0967520b0c35ce05"}
Oct 30 23:43:19.713620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:19.713574    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz"
Oct 30 23:43:19.811334 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:19.811304    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09f50622-c9d6-43c2-bde6-219e11207233-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f50622-c9d6-43c2-bde6-219e11207233-maas-api-openshift-ca\") pod \"09f50622-c9d6-43c2-bde6-219e11207233\" (UID: \"09f50622-c9d6-43c2-bde6-219e11207233\") "
Oct 30 23:43:19.811528 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:19.811508    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09f50622-c9d6-43c2-bde6-219e11207233-kube-api-access-djdp7\" (UniqueName: \"kubernetes.io/projected/09f50622-c9d6-43c2-bde6-219e11207233-kube-api-access-djdp7\") pod \"09f50622-c9d6-43c2-bde6-219e11207233\" (UID: \"09f50622-c9d6-43c2-bde6-219e11207233\") "
Oct 30 23:43:19.811668 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:19.811650    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09f50622-c9d6-43c2-bde6-219e11207233-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f50622-c9d6-43c2-bde6-219e11207233-maas-api-xks-ca\") pod \"09f50622-c9d6-43c2-bde6-219e11207233\" (UID: \"09f50622-c9d6-43c2-bde6-219e11207233\") "
Oct 30 23:43:19.811734 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:19.811711    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09f50622-c9d6-43c2-bde6-219e11207233-maas-api-openshift-ca" pod "09f50622-c9d6-43c2-bde6-219e11207233" (UID: "09f50622-c9d6-43c2-bde6-219e11207233"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:19.812115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:19.812087    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09f50622-c9d6-43c2-bde6-219e11207233-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:19.813765 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:19.813742    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09f50622-c9d6-43c2-bde6-219e11207233-maas-api-xks-ca" pod "09f50622-c9d6-43c2-bde6-219e11207233" (UID: "09f50622-c9d6-43c2-bde6-219e11207233"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:19.813858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:19.813776    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09f50622-c9d6-43c2-bde6-219e11207233-kube-api-access-djdp7" pod "09f50622-c9d6-43c2-bde6-219e11207233" (UID: "09f50622-c9d6-43c2-bde6-219e11207233"). InnerVolumeSpecName "kube-api-access-djdp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:19.913295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:19.913245    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djdp7\" (UniqueName: \"kubernetes.io/projected/09f50622-c9d6-43c2-bde6-219e11207233-kube-api-access-djdp7\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:19.913295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:19.913275    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09f50622-c9d6-43c2-bde6-219e11207233-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:20.517330 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:20.517299    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz"
Oct 30 23:43:20.517522 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:20.517341    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz" event={"ID":"09f50622-c9d6-43c2-bde6-219e11207233","Type":"ContainerDied","Data":"0dbcac3def57d8b7776a7a2bfa5f298aaa46a39dfe8fbf319ae66e5db5401ab4"}
Oct 30 23:43:20.517522 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:20.517430    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dbcac3def57d8b7776a7a2bfa5f298aaa46a39dfe8fbf319ae66e5db5401ab4"
Oct 30 23:43:42.763805 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.763773    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"]
Oct 30 23:43:42.764400 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.764385    5078 cpu_manager.go:414] "containerMap: removing container" podUID="09f50622-c9d6-43c2-bde6-219e11207233" containerName="revoke-keys"
Oct 30 23:43:42.764446 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.764402    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09f50622-c9d6-43c2-bde6-219e11207233" containerName="revoke-keys"
Oct 30 23:43:42.764514 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.764504    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="09f50622-c9d6-43c2-bde6-219e11207233" containerName="revoke-keys"
Oct 30 23:43:42.769101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.769085    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.813622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.813566    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"]
Oct 30 23:43:42.817605 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.817581    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.865400 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.865366    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"]
Oct 30 23:43:42.889536 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.889504    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.889758 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.889546    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d729\" (UniqueName: \"kubernetes.io/projected/7ceff1f9-9fc7-4a93-9619-1605386e0b64-kube-api-access-6d729\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.889758 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.889573    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.889758 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.889730    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.889989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.889790    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.889989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.889844    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.889989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.889893    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.889989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.889916    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.889989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.889971    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.890268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.890034    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.890268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.890081    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.890268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.890101    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313b3ad0-1127-40c2-85b3-4a1555935650-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.890268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.890116    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.890268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.890239    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313b3ad0-1127-40c2-85b3-4a1555935650-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.890544 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.890319    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313b3ad0-1127-40c2-85b3-4a1555935650-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.890544 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.890385    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.890544 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.890462    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kwgz\" (UniqueName: \"kubernetes.io/projected/313b3ad0-1127-40c2-85b3-4a1555935650-kube-api-access-6kwgz\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.890544 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.890533    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.909327 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.909292    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"]
Oct 30 23:43:42.991542 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.991511    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.991768 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.991567    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.991768 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.991627    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.991768 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.991658    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.991768 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.991693    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.991768 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.991718    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.991768 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.991751    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.992128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.991819    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.992128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.991877    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.992128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.991905    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313b3ad0-1127-40c2-85b3-4a1555935650-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.992128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.991930    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.992128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.991978    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313b3ad0-1127-40c2-85b3-4a1555935650-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.992128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.992008    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.992128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.992028    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.992128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.992068    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313b3ad0-1127-40c2-85b3-4a1555935650-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.992128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.992106    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.992662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.992142    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.992662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.992175    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kwgz\" (UniqueName: \"kubernetes.io/projected/313b3ad0-1127-40c2-85b3-4a1555935650-kube-api-access-6kwgz\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.992662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.992250    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-istio-data\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.992662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.992255    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.992662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.992326    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.992662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.992365    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-workload-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.992662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.992373    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d729\" (UniqueName: \"kubernetes.io/projected/7ceff1f9-9fc7-4a93-9619-1605386e0b64-kube-api-access-6d729\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.992662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.992522    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.992662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.992586    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-credential-socket\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.993165 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.992859    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-workload-certs\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.993276 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.993254    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313b3ad0-1127-40c2-85b3-4a1555935650-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.993345 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.993296    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istiod-ca-cert\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.994504 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.994483    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313b3ad0-1127-40c2-85b3-4a1555935650-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.994748 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.994730    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:42.995145 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.995125    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-envoy\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:42.995468 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:42.995444    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-podinfo\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:43.061677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.061645    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:43.062953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.062930    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kwgz\" (UniqueName: \"kubernetes.io/projected/313b3ad0-1127-40c2-85b3-4a1555935650-kube-api-access-6kwgz\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:43.063227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.063206    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313b3ad0-1127-40c2-85b3-4a1555935650-istio-token\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-99qzp\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:43.063364 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.063347    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d729\" (UniqueName: \"kubernetes.io/projected/7ceff1f9-9fc7-4a93-9619-1605386e0b64-kube-api-access-6d729\") pod \"e2e-mt-55666687-openshift-default-5748f754cc-sp467\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") " pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:43.082288 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.082266    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:43.130976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.130941    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:43.518357 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.518329    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"]
Oct 30 23:43:43.518622 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:43:43.518568    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ceff1f9_9fc7_4a93_9619_1605386e0b64.slice/crio-5da5f5b6ef02458ca1197803db97770100cc6f4661201af041df9e5a7987f7b7 WatchSource:0}: Error finding container 5da5f5b6ef02458ca1197803db97770100cc6f4661201af041df9e5a7987f7b7: Status 404 returned error can't find the container with id 5da5f5b6ef02458ca1197803db97770100cc6f4661201af041df9e5a7987f7b7
Oct 30 23:43:43.521826 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.521793    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:43:43.521915 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.521873    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:43:43.521961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.521916    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:43:43.571151 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.571120    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"]
Oct 30 23:43:43.581848 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:43:43.581824    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313b3ad0_1127_40c2_85b3_4a1555935650.slice/crio-ed8db85eba9d810936ec40b6b1070ad480fbf7a1d2d6ec91862876d9a48f84d5 WatchSource:0}: Error finding container ed8db85eba9d810936ec40b6b1070ad480fbf7a1d2d6ec91862876d9a48f84d5: Status 404 returned error can't find the container with id ed8db85eba9d810936ec40b6b1070ad480fbf7a1d2d6ec91862876d9a48f84d5
Oct 30 23:43:43.585286 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.585249    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:43:43.585372 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.585341    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:43:43.585413 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.585390    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:43:43.682857 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.682149    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp" event={"ID":"313b3ad0-1127-40c2-85b3-4a1555935650","Type":"ContainerStarted","Data":"2a14e1ae6108ae85129fd8a201c3c45016208cdcde970af0cedd71c52e38c866"}
Oct 30 23:43:43.682857 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.682213    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp" event={"ID":"313b3ad0-1127-40c2-85b3-4a1555935650","Type":"ContainerStarted","Data":"ed8db85eba9d810936ec40b6b1070ad480fbf7a1d2d6ec91862876d9a48f84d5"}
Oct 30 23:43:43.684840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.684807    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467" event={"ID":"7ceff1f9-9fc7-4a93-9619-1605386e0b64","Type":"ContainerStarted","Data":"f99911c9c866e276cdf24ea346b7cc8d1008c6a5e57436b62b331221a5f4eee1"}
Oct 30 23:43:43.684979 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.684866    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467" event={"ID":"7ceff1f9-9fc7-4a93-9619-1605386e0b64","Type":"ContainerStarted","Data":"5da5f5b6ef02458ca1197803db97770100cc6f4661201af041df9e5a7987f7b7"}
Oct 30 23:43:43.808056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.807995    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp" podStartSLOduration=1.8079805439999999 podStartE2EDuration="1.807980544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:43.805586537 +0000 UTC m=+2209.693492975" watchObservedRunningTime="2026-10-30 23:43:43.807980544 +0000 UTC m=+2209.695886971"
Oct 30 23:43:43.928870 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:43.928809    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467" podStartSLOduration=1.928789949 podStartE2EDuration="1.928789949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:43.924400228 +0000 UTC m=+2209.812306582" watchObservedRunningTime="2026-10-30 23:43:43.928789949 +0000 UTC m=+2209.816696305"
Oct 30 23:43:44.083070 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:44.082926    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:44.086576 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:44.086542    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-sp467 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 23:43:44.086868 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:44.086838    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467" podUID="7ceff1f9-9fc7-4a93-9619-1605386e0b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 23:43:44.131928 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:44.131898    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:44.133650 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:44.133625    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-99qzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 23:43:44.133778 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:44.133669    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp" podUID="313b3ad0-1127-40c2-85b3-4a1555935650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 23:43:45.084621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:45.084018    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-sp467 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 23:43:45.084621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:45.084086    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467" podUID="7ceff1f9-9fc7-4a93-9619-1605386e0b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 23:43:45.131853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:45.131820    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-99qzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 23:43:45.132032 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:45.131871    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp" podUID="313b3ad0-1127-40c2-85b3-4a1555935650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 23:43:46.104035 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:46.103576    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-sp467 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 23:43:46.104035 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:46.103666    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467" podUID="7ceff1f9-9fc7-4a93-9619-1605386e0b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 23:43:46.132833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:46.131732    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-99qzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 23:43:46.132833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:46.131869    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp" podUID="313b3ad0-1127-40c2-85b3-4a1555935650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 23:43:47.093624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.092999    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-sp467 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 23:43:47.093624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.093058    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467" podUID="7ceff1f9-9fc7-4a93-9619-1605386e0b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 23:43:47.131893 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.131858    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-99qzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 23:43:47.132281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.131907    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp" podUID="313b3ad0-1127-40c2-85b3-4a1555935650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 23:43:47.636189 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:43:47.636160    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:43:47.813798 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.813763    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg"]
Oct 30 23:43:47.817949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.817927    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg"
Oct 30 23:43:47.823771 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.823744    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-55666687-5jplg\" (UID: \"7bfd60b4-9b7c-41c2-ba66-a60640cf554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg"
Oct 30 23:43:47.823929 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.823910    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69d95\" (UniqueName: \"kubernetes.io/projected/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-kube-api-access-69d95\") pod \"maas-api-revoke-keys-e2e-mt-55666687-5jplg\" (UID: \"7bfd60b4-9b7c-41c2-ba66-a60640cf554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg"
Oct 30 23:43:47.824044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.824030    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-55666687-5jplg\" (UID: \"7bfd60b4-9b7c-41c2-ba66-a60640cf554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg"
Oct 30 23:43:47.888577 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.888485    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg"]
Oct 30 23:43:47.923022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.922873    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:43:47.923022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.922921    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:43:47.923022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.922923    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:43:47.923022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.923018    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:43:47.927526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.927496    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69d95\" (UniqueName: \"kubernetes.io/projected/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-kube-api-access-69d95\") pod \"maas-api-revoke-keys-e2e-mt-55666687-5jplg\" (UID: \"7bfd60b4-9b7c-41c2-ba66-a60640cf554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg"
Oct 30 23:43:47.927714 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.927679    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-55666687-5jplg\" (UID: \"7bfd60b4-9b7c-41c2-ba66-a60640cf554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg"
Oct 30 23:43:47.927833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.927809    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-55666687-5jplg\" (UID: \"7bfd60b4-9b7c-41c2-ba66-a60640cf554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg"
Oct 30 23:43:47.928890 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.928868    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-55666687-5jplg\" (UID: \"7bfd60b4-9b7c-41c2-ba66-a60640cf554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg"
Oct 30 23:43:47.932064 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:47.932041    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-55666687-5jplg\" (UID: \"7bfd60b4-9b7c-41c2-ba66-a60640cf554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg"
Oct 30 23:43:48.010707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:48.010671    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69d95\" (UniqueName: \"kubernetes.io/projected/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-kube-api-access-69d95\") pod \"maas-api-revoke-keys-e2e-mt-55666687-5jplg\" (UID: \"7bfd60b4-9b7c-41c2-ba66-a60640cf554e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg"
Oct 30 23:43:48.086376 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:48.085920    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-sp467 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 23:43:48.086376 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:48.085978    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467" podUID="7ceff1f9-9fc7-4a93-9619-1605386e0b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 23:43:48.131474 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:48.131445    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-99qzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 23:43:48.131728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:48.131493    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp" podUID="313b3ad0-1127-40c2-85b3-4a1555935650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 23:43:48.132646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:48.132618    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg"
Oct 30 23:43:48.612625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:48.600286    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg"]
Oct 30 23:43:48.612625 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:43:48.609080    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bfd60b4_9b7c_41c2_ba66_a60640cf554e.slice/crio-b5f41a9b53d9c9b0db9874ff33f0f460f3e8edc4713e5c8bd8684822c9893b7c WatchSource:0}: Error finding container b5f41a9b53d9c9b0db9874ff33f0f460f3e8edc4713e5c8bd8684822c9893b7c: Status 404 returned error can't find the container with id b5f41a9b53d9c9b0db9874ff33f0f460f3e8edc4713e5c8bd8684822c9893b7c
Oct 30 23:43:48.739413 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:48.739374    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg" event={"ID":"7bfd60b4-9b7c-41c2-ba66-a60640cf554e","Type":"ContainerStarted","Data":"b5f41a9b53d9c9b0db9874ff33f0f460f3e8edc4713e5c8bd8684822c9893b7c"}
Oct 30 23:43:49.090031 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:49.089987    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-sp467 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused" start-of-body=
Oct 30 23:43:49.090228 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:49.090054    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467" podUID="7ceff1f9-9fc7-4a93-9619-1605386e0b64" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.191:15021/healthz/ready\": dial tcp 10.133.0.191:15021: connect: connection refused"
Oct 30 23:43:49.131522 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:49.131490    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-99qzp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused" start-of-body=
Oct 30 23:43:49.131663 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:49.131562    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp" podUID="313b3ad0-1127-40c2-85b3-4a1555935650" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.192:15021/healthz/ready\": dial tcp 10.133.0.192:15021: connect: connection refused"
Oct 30 23:43:49.755692 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:49.755652    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg" event={"ID":"7bfd60b4-9b7c-41c2-ba66-a60640cf554e","Type":"ContainerStarted","Data":"d574d219b312901b4d05007261161e594ea137637924edb53d7682b33987e977"}
Oct 30 23:43:49.872756 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:49.872532    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg" podStartSLOduration=2.8725179880000002 podStartE2EDuration="2.872517988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:43:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:43:49.870014856 +0000 UTC m=+2215.757921210" watchObservedRunningTime="2026-10-30 23:43:49.872517988 +0000 UTC m=+2215.760424376"
Oct 30 23:43:50.135195 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:50.135168    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:50.135904 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:50.135872    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:50.136682 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:50.136663    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:43:50.152097 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:50.152074    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:50.152411 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:50.152387    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:50.153252 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:50.153235    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:43:50.765032 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:50.765000    5078 generic.go:356] "Generic (PLEG): container finished" podID="7bfd60b4-9b7c-41c2-ba66-a60640cf554e" containerID="d574d219b312901b4d05007261161e594ea137637924edb53d7682b33987e977" exitCode=0
Oct 30 23:43:50.765436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:50.765086    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg" event={"ID":"7bfd60b4-9b7c-41c2-ba66-a60640cf554e","Type":"ContainerDied","Data":"d574d219b312901b4d05007261161e594ea137637924edb53d7682b33987e977"}
Oct 30 23:43:51.937713 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:51.937688    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg"
Oct 30 23:43:51.986908 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:51.986885    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-maas-api-openshift-ca\") pod \"7bfd60b4-9b7c-41c2-ba66-a60640cf554e\" (UID: \"7bfd60b4-9b7c-41c2-ba66-a60640cf554e\") "
Oct 30 23:43:51.987039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:51.986990    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-maas-api-xks-ca\") pod \"7bfd60b4-9b7c-41c2-ba66-a60640cf554e\" (UID: \"7bfd60b4-9b7c-41c2-ba66-a60640cf554e\") "
Oct 30 23:43:51.987079 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:51.987060    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-kube-api-access-69d95\" (UniqueName: \"kubernetes.io/projected/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-kube-api-access-69d95\") pod \"7bfd60b4-9b7c-41c2-ba66-a60640cf554e\" (UID: \"7bfd60b4-9b7c-41c2-ba66-a60640cf554e\") "
Oct 30 23:43:51.987228 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:51.987210    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-maas-api-openshift-ca" pod "7bfd60b4-9b7c-41c2-ba66-a60640cf554e" (UID: "7bfd60b4-9b7c-41c2-ba66-a60640cf554e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:43:51.987566 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:51.987535    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:51.989205 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:51.989182    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-maas-api-xks-ca" pod "7bfd60b4-9b7c-41c2-ba66-a60640cf554e" (UID: "7bfd60b4-9b7c-41c2-ba66-a60640cf554e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:43:51.989305 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:51.989278    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-kube-api-access-69d95" pod "7bfd60b4-9b7c-41c2-ba66-a60640cf554e" (UID: "7bfd60b4-9b7c-41c2-ba66-a60640cf554e"). InnerVolumeSpecName "kube-api-access-69d95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:43:52.089245 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:52.089210    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:52.089417 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:52.089250    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69d95\" (UniqueName: \"kubernetes.io/projected/7bfd60b4-9b7c-41c2-ba66-a60640cf554e-kube-api-access-69d95\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:43:52.780310 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:52.780284    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg"
Oct 30 23:43:52.780488 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:52.780281    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg" event={"ID":"7bfd60b4-9b7c-41c2-ba66-a60640cf554e","Type":"ContainerDied","Data":"b5f41a9b53d9c9b0db9874ff33f0f460f3e8edc4713e5c8bd8684822c9893b7c"}
Oct 30 23:43:52.780488 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:52.780409    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5f41a9b53d9c9b0db9874ff33f0f460f3e8edc4713e5c8bd8684822c9893b7c"
Oct 30 23:43:57.108682 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:43:57.108647    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs"]
Oct 30 23:44:02.109258 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:02.109187    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs" podUID="e0307507-cd4c-4129-96ec-9005d468f73b" containerName="payload-processing" containerID="cri-o://ea956851a9c2473d70780cffece5dc617645f4927f2a00b3e8e93f458419ab18" gracePeriod=25
Oct 30 23:44:02.411240 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:02.411214    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs"
Oct 30 23:44:02.499261 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:02.499229    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0307507-cd4c-4129-96ec-9005d468f73b-kube-api-access-kmskv\" (UniqueName: \"kubernetes.io/projected/e0307507-cd4c-4129-96ec-9005d468f73b-kube-api-access-kmskv\") pod \"e0307507-cd4c-4129-96ec-9005d468f73b\" (UID: \"e0307507-cd4c-4129-96ec-9005d468f73b\") "
Oct 30 23:44:02.499451 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:02.499326    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0307507-cd4c-4129-96ec-9005d468f73b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0307507-cd4c-4129-96ec-9005d468f73b-plugins-config-volume\") pod \"e0307507-cd4c-4129-96ec-9005d468f73b\" (UID: \"e0307507-cd4c-4129-96ec-9005d468f73b\") "
Oct 30 23:44:02.499750 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:02.499726    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0307507-cd4c-4129-96ec-9005d468f73b-plugins-config-volume" pod "e0307507-cd4c-4129-96ec-9005d468f73b" (UID: "e0307507-cd4c-4129-96ec-9005d468f73b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:44:02.499917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:02.499903    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0307507-cd4c-4129-96ec-9005d468f73b-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:02.501688 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:02.501667    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0307507-cd4c-4129-96ec-9005d468f73b-kube-api-access-kmskv" pod "e0307507-cd4c-4129-96ec-9005d468f73b" (UID: "e0307507-cd4c-4129-96ec-9005d468f73b"). InnerVolumeSpecName "kube-api-access-kmskv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:02.601344 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:02.601308    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmskv\" (UniqueName: \"kubernetes.io/projected/e0307507-cd4c-4129-96ec-9005d468f73b-kube-api-access-kmskv\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:02.986584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:02.986551    5078 generic.go:356] "Generic (PLEG): container finished" podID="e0307507-cd4c-4129-96ec-9005d468f73b" containerID="ea956851a9c2473d70780cffece5dc617645f4927f2a00b3e8e93f458419ab18" exitCode=0
Oct 30 23:44:02.986792 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:02.986633    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs"
Oct 30 23:44:02.986792 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:02.986642    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs" event={"ID":"e0307507-cd4c-4129-96ec-9005d468f73b","Type":"ContainerDied","Data":"ea956851a9c2473d70780cffece5dc617645f4927f2a00b3e8e93f458419ab18"}
Oct 30 23:44:02.986792 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:02.986702    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs" event={"ID":"e0307507-cd4c-4129-96ec-9005d468f73b","Type":"ContainerDied","Data":"0f45bdf37c9686bc5f18df41be79932e65442bdcffdcd622ea124172ca000a3c"}
Oct 30 23:44:02.986792 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:02.986726    5078 scope.go:122] "RemoveContainer" containerID="ea956851a9c2473d70780cffece5dc617645f4927f2a00b3e8e93f458419ab18"
Oct 30 23:44:03.012513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:03.012489    5078 scope.go:122] "RemoveContainer" containerID="ea956851a9c2473d70780cffece5dc617645f4927f2a00b3e8e93f458419ab18"
Oct 30 23:44:03.012830 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:44:03.012810    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea956851a9c2473d70780cffece5dc617645f4927f2a00b3e8e93f458419ab18\": container with ID starting with ea956851a9c2473d70780cffece5dc617645f4927f2a00b3e8e93f458419ab18 not found: ID does not exist" containerID="ea956851a9c2473d70780cffece5dc617645f4927f2a00b3e8e93f458419ab18"
Oct 30 23:44:03.012904 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:03.012843    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea956851a9c2473d70780cffece5dc617645f4927f2a00b3e8e93f458419ab18"} err="failed to get container status \"ea956851a9c2473d70780cffece5dc617645f4927f2a00b3e8e93f458419ab18\": rpc error: code = NotFound desc = could not find container \"ea956851a9c2473d70780cffece5dc617645f4927f2a00b3e8e93f458419ab18\": container with ID starting with ea956851a9c2473d70780cffece5dc617645f4927f2a00b3e8e93f458419ab18 not found: ID does not exist"
Oct 30 23:44:03.155796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:03.155765    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs"]
Oct 30 23:44:03.168177 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:03.168149    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-55666687-cdc987d69-km7zs"]
Oct 30 23:44:03.434620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:03.420654    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg"]
Oct 30 23:44:03.480308 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:03.471259    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-196ff7-k9vlg"]
Oct 30 23:44:04.298883 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:04.298854    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29c46b1e-954c-4c1d-a709-6044a85b855a" path="/var/lib/kubelet/pods/29c46b1e-954c-4c1d-a709-6044a85b855a/volumes"
Oct 30 23:44:04.299286 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:04.299271    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0307507-cd4c-4129-96ec-9005d468f73b" path="/var/lib/kubelet/pods/e0307507-cd4c-4129-96ec-9005d468f73b/volumes"
Oct 30 23:44:04.843381 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:04.843352    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp"]
Oct 30 23:44:04.891255 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:04.891225    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-7a42d8-q8kqp"]
Oct 30 23:44:06.298937 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:06.298903    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c4f5f2-82ce-47a5-bfcf-a31feb269947" path="/var/lib/kubelet/pods/63c4f5f2-82ce-47a5-bfcf-a31feb269947/volumes"
Oct 30 23:44:08.047201 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:08.047166    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx"]
Oct 30 23:44:08.096773 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:08.096741    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-721fec8b-nmplx"]
Oct 30 23:44:08.298772 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:08.298690    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="472179d6-45c3-4598-8961-23c2f46b6fea" path="/var/lib/kubelet/pods/472179d6-45c3-4598-8961-23c2f46b6fea/volumes"
Oct 30 23:44:10.114350 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:10.114321    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"]
Oct 30 23:44:10.114803 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:10.114532    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r" podUID="fc173733-80d8-4793-b1a1-723844b99673" containerName="istio-proxy" containerID="cri-o://4149f24d3cc318e962f2c40a15f315175b0a27b626727cfc1f3571cd2f6f0fad" gracePeriod=30
Oct 30 23:44:10.982104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:10.982073    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5"]
Oct 30 23:44:10.982664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:10.982650    5078 cpu_manager.go:414] "containerMap: removing container" podUID="e0307507-cd4c-4129-96ec-9005d468f73b" containerName="payload-processing"
Oct 30 23:44:10.982741 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:10.982665    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0307507-cd4c-4129-96ec-9005d468f73b" containerName="payload-processing"
Oct 30 23:44:10.982741 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:10.982680    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7bfd60b4-9b7c-41c2-ba66-a60640cf554e" containerName="revoke-keys"
Oct 30 23:44:10.982741 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:10.982684    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bfd60b4-9b7c-41c2-ba66-a60640cf554e" containerName="revoke-keys"
Oct 30 23:44:10.982856 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:10.982783    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0307507-cd4c-4129-96ec-9005d468f73b" containerName="payload-processing"
Oct 30 23:44:10.982856 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:10.982792    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bfd60b4-9b7c-41c2-ba66-a60640cf554e" containerName="revoke-keys"
Oct 30 23:44:10.987572 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:10.987553    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5"
Oct 30 23:44:11.040898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:11.040248    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:44:11.040898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:11.040563    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:44:11.071421 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:11.071381    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5"]
Oct 30 23:44:11.089651 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:11.089583    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:44:11.089881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:11.089674    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:44:11.101659 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:11.101634    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj7wb\" (UniqueName: \"kubernetes.io/projected/66fc5643-d66c-4cb5-98c7-15af84bf7167-kube-api-access-vj7wb\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5\" (UID: \"66fc5643-d66c-4cb5-98c7-15af84bf7167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5"
Oct 30 23:44:11.101856 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:11.101841    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66fc5643-d66c-4cb5-98c7-15af84bf7167-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5\" (UID: \"66fc5643-d66c-4cb5-98c7-15af84bf7167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5"
Oct 30 23:44:11.101923 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:11.101890    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66fc5643-d66c-4cb5-98c7-15af84bf7167-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5\" (UID: \"66fc5643-d66c-4cb5-98c7-15af84bf7167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5"
Oct 30 23:44:11.202947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:11.202918    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj7wb\" (UniqueName: \"kubernetes.io/projected/66fc5643-d66c-4cb5-98c7-15af84bf7167-kube-api-access-vj7wb\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5\" (UID: \"66fc5643-d66c-4cb5-98c7-15af84bf7167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5"
Oct 30 23:44:11.203347 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:11.203148    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66fc5643-d66c-4cb5-98c7-15af84bf7167-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5\" (UID: \"66fc5643-d66c-4cb5-98c7-15af84bf7167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5"
Oct 30 23:44:11.203347 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:11.203196    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66fc5643-d66c-4cb5-98c7-15af84bf7167-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5\" (UID: \"66fc5643-d66c-4cb5-98c7-15af84bf7167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5"
Oct 30 23:44:11.204043 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:11.203968    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66fc5643-d66c-4cb5-98c7-15af84bf7167-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5\" (UID: \"66fc5643-d66c-4cb5-98c7-15af84bf7167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5"
Oct 30 23:44:11.205728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:11.205709    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66fc5643-d66c-4cb5-98c7-15af84bf7167-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5\" (UID: \"66fc5643-d66c-4cb5-98c7-15af84bf7167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5"
Oct 30 23:44:11.263990 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:11.263936    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj7wb\" (UniqueName: \"kubernetes.io/projected/66fc5643-d66c-4cb5-98c7-15af84bf7167-kube-api-access-vj7wb\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5\" (UID: \"66fc5643-d66c-4cb5-98c7-15af84bf7167\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5"
Oct 30 23:44:11.300984 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:11.300961    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5"
Oct 30 23:44:11.684246 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:11.684220    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5"]
Oct 30 23:44:11.685022 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:44:11.684994    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66fc5643_d66c_4cb5_98c7_15af84bf7167.slice/crio-1cf0c47d30aca4a33c486fff44cb9c3d7ead75f5dc890266607ced5f97af3799 WatchSource:0}: Error finding container 1cf0c47d30aca4a33c486fff44cb9c3d7ead75f5dc890266607ced5f97af3799: Status 404 returned error can't find the container with id 1cf0c47d30aca4a33c486fff44cb9c3d7ead75f5dc890266607ced5f97af3799
Oct 30 23:44:12.056148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:12.056112    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5" event={"ID":"66fc5643-d66c-4cb5-98c7-15af84bf7167","Type":"ContainerStarted","Data":"1655d3c31cdfef6416e344ce6ad55cd0fdd03a5fd106132adf00815f85cd053b"}
Oct 30 23:44:12.056322 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:12.056175    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5" event={"ID":"66fc5643-d66c-4cb5-98c7-15af84bf7167","Type":"ContainerStarted","Data":"1cf0c47d30aca4a33c486fff44cb9c3d7ead75f5dc890266607ced5f97af3799"}
Oct 30 23:44:12.172225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:12.172176    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5" podStartSLOduration=2.172160975 podStartE2EDuration="2.172160975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:44:12.170411293 +0000 UTC m=+2238.058317646" watchObservedRunningTime="2026-10-30 23:44:12.172160975 +0000 UTC m=+2238.060067328"
Oct 30 23:44:12.554912 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:12.554882    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"]
Oct 30 23:44:12.560315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:12.560284    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv" podUID="4cf62568-28ad-43c6-a0ab-f1680ef1a72c" containerName="istio-proxy" containerID="cri-o://7f56c1761fcc608527aed38a6bc595c602d514bce08473de1db7effb2cd8c077" gracePeriod=30
Oct 30 23:44:13.064730 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:13.064695    5078 generic.go:356] "Generic (PLEG): container finished" podID="66fc5643-d66c-4cb5-98c7-15af84bf7167" containerID="1655d3c31cdfef6416e344ce6ad55cd0fdd03a5fd106132adf00815f85cd053b" exitCode=0
Oct 30 23:44:13.064918 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:13.064782    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5" event={"ID":"66fc5643-d66c-4cb5-98c7-15af84bf7167","Type":"ContainerDied","Data":"1655d3c31cdfef6416e344ce6ad55cd0fdd03a5fd106132adf00815f85cd053b"}
Oct 30 23:44:14.385711 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:14.385687    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5"
Oct 30 23:44:14.475132 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:14.475101    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66fc5643-d66c-4cb5-98c7-15af84bf7167-kube-api-access-vj7wb\" (UniqueName: \"kubernetes.io/projected/66fc5643-d66c-4cb5-98c7-15af84bf7167-kube-api-access-vj7wb\") pod \"66fc5643-d66c-4cb5-98c7-15af84bf7167\" (UID: \"66fc5643-d66c-4cb5-98c7-15af84bf7167\") "
Oct 30 23:44:14.475336 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:14.475314    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66fc5643-d66c-4cb5-98c7-15af84bf7167-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66fc5643-d66c-4cb5-98c7-15af84bf7167-maas-api-xks-ca\") pod \"66fc5643-d66c-4cb5-98c7-15af84bf7167\" (UID: \"66fc5643-d66c-4cb5-98c7-15af84bf7167\") "
Oct 30 23:44:14.475411 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:14.475368    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66fc5643-d66c-4cb5-98c7-15af84bf7167-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66fc5643-d66c-4cb5-98c7-15af84bf7167-maas-api-openshift-ca\") pod \"66fc5643-d66c-4cb5-98c7-15af84bf7167\" (UID: \"66fc5643-d66c-4cb5-98c7-15af84bf7167\") "
Oct 30 23:44:14.475777 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:14.475751    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66fc5643-d66c-4cb5-98c7-15af84bf7167-maas-api-openshift-ca" pod "66fc5643-d66c-4cb5-98c7-15af84bf7167" (UID: "66fc5643-d66c-4cb5-98c7-15af84bf7167"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:44:14.475955 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:14.475932    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66fc5643-d66c-4cb5-98c7-15af84bf7167-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:14.477383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:14.477361    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66fc5643-d66c-4cb5-98c7-15af84bf7167-kube-api-access-vj7wb" pod "66fc5643-d66c-4cb5-98c7-15af84bf7167" (UID: "66fc5643-d66c-4cb5-98c7-15af84bf7167"). InnerVolumeSpecName "kube-api-access-vj7wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:14.477644 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:14.477620    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66fc5643-d66c-4cb5-98c7-15af84bf7167-maas-api-xks-ca" pod "66fc5643-d66c-4cb5-98c7-15af84bf7167" (UID: "66fc5643-d66c-4cb5-98c7-15af84bf7167"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:44:14.577328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:14.577290    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj7wb\" (UniqueName: \"kubernetes.io/projected/66fc5643-d66c-4cb5-98c7-15af84bf7167-kube-api-access-vj7wb\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:14.577328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:14.577331    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66fc5643-d66c-4cb5-98c7-15af84bf7167-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:15.104777 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.104740    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5"
Oct 30 23:44:15.104969 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.104793    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5" event={"ID":"66fc5643-d66c-4cb5-98c7-15af84bf7167","Type":"ContainerDied","Data":"1cf0c47d30aca4a33c486fff44cb9c3d7ead75f5dc890266607ced5f97af3799"}
Oct 30 23:44:15.104969 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.104903    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cf0c47d30aca4a33c486fff44cb9c3d7ead75f5dc890266607ced5f97af3799"
Oct 30 23:44:15.375451 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.375429    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:44:15.501568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.501533    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-istio-envoy\") pod \"fc173733-80d8-4793-b1a1-723844b99673\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") "
Oct 30 23:44:15.502077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.501660    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc173733-80d8-4793-b1a1-723844b99673-kube-api-access-7qbqg\" (UniqueName: \"kubernetes.io/projected/fc173733-80d8-4793-b1a1-723844b99673-kube-api-access-7qbqg\") pod \"fc173733-80d8-4793-b1a1-723844b99673\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") "
Oct 30 23:44:15.502077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.501695    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-istio-data\") pod \"fc173733-80d8-4793-b1a1-723844b99673\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") "
Oct 30 23:44:15.502077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.501724    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-workload-certs\") pod \"fc173733-80d8-4793-b1a1-723844b99673\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") "
Oct 30 23:44:15.502077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.501744    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc173733-80d8-4793-b1a1-723844b99673-istio-token\" (UniqueName: \"kubernetes.io/projected/fc173733-80d8-4793-b1a1-723844b99673-istio-token\") pod \"fc173733-80d8-4793-b1a1-723844b99673\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") "
Oct 30 23:44:15.502077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.501851    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc173733-80d8-4793-b1a1-723844b99673-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc173733-80d8-4793-b1a1-723844b99673-istio-podinfo\") pod \"fc173733-80d8-4793-b1a1-723844b99673\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") "
Oct 30 23:44:15.502077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.501907    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-credential-socket\") pod \"fc173733-80d8-4793-b1a1-723844b99673\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") "
Oct 30 23:44:15.502077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.501984    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc173733-80d8-4793-b1a1-723844b99673-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc173733-80d8-4793-b1a1-723844b99673-istiod-ca-cert\") pod \"fc173733-80d8-4793-b1a1-723844b99673\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") "
Oct 30 23:44:15.502077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.501974    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-workload-certs" pod "fc173733-80d8-4793-b1a1-723844b99673" (UID: "fc173733-80d8-4793-b1a1-723844b99673"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:15.502077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.502011    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-workload-socket\") pod \"fc173733-80d8-4793-b1a1-723844b99673\" (UID: \"fc173733-80d8-4793-b1a1-723844b99673\") "
Oct 30 23:44:15.502077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.502046    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-istio-data" pod "fc173733-80d8-4793-b1a1-723844b99673" (UID: "fc173733-80d8-4793-b1a1-723844b99673"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:15.502835 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.502199    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-credential-socket" pod "fc173733-80d8-4793-b1a1-723844b99673" (UID: "fc173733-80d8-4793-b1a1-723844b99673"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:15.502835 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.502383    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc173733-80d8-4793-b1a1-723844b99673-istiod-ca-cert" pod "fc173733-80d8-4793-b1a1-723844b99673" (UID: "fc173733-80d8-4793-b1a1-723844b99673"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:44:15.502835 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.502398    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-workload-socket" pod "fc173733-80d8-4793-b1a1-723844b99673" (UID: "fc173733-80d8-4793-b1a1-723844b99673"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:15.503007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.502866    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:15.503007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.502892    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc173733-80d8-4793-b1a1-723844b99673-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:15.503007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.502909    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:15.503007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.502924    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:15.503007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.502939    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:15.504135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.504111    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-istio-envoy" pod "fc173733-80d8-4793-b1a1-723844b99673" (UID: "fc173733-80d8-4793-b1a1-723844b99673"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:15.504135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.504127    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc173733-80d8-4793-b1a1-723844b99673-kube-api-access-7qbqg" pod "fc173733-80d8-4793-b1a1-723844b99673" (UID: "fc173733-80d8-4793-b1a1-723844b99673"). InnerVolumeSpecName "kube-api-access-7qbqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:15.504289 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.504164    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc173733-80d8-4793-b1a1-723844b99673-istio-token" pod "fc173733-80d8-4793-b1a1-723844b99673" (UID: "fc173733-80d8-4793-b1a1-723844b99673"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:15.504289 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.504169    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc173733-80d8-4793-b1a1-723844b99673-istio-podinfo" pod "fc173733-80d8-4793-b1a1-723844b99673" (UID: "fc173733-80d8-4793-b1a1-723844b99673"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:44:15.651687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.618564    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc173733-80d8-4793-b1a1-723844b99673-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:15.651687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.618621    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qbqg\" (UniqueName: \"kubernetes.io/projected/fc173733-80d8-4793-b1a1-723844b99673-kube-api-access-7qbqg\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:15.651687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.618638    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc173733-80d8-4793-b1a1-723844b99673-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:15.651687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:15.618653    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc173733-80d8-4793-b1a1-723844b99673-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:16.113759 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:16.113725    5078 generic.go:356] "Generic (PLEG): container finished" podID="fc173733-80d8-4793-b1a1-723844b99673" containerID="4149f24d3cc318e962f2c40a15f315175b0a27b626727cfc1f3571cd2f6f0fad" exitCode=0
Oct 30 23:44:16.113954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:16.113795    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"
Oct 30 23:44:16.113954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:16.113810    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r" event={"ID":"fc173733-80d8-4793-b1a1-723844b99673","Type":"ContainerDied","Data":"4149f24d3cc318e962f2c40a15f315175b0a27b626727cfc1f3571cd2f6f0fad"}
Oct 30 23:44:16.113954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:16.113885    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r" event={"ID":"fc173733-80d8-4793-b1a1-723844b99673","Type":"ContainerDied","Data":"e052506ba69888d254476d1df6b7215028d490b3a9deb2bcf8e97fad0c3f9835"}
Oct 30 23:44:16.113954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:16.113905    5078 scope.go:122] "RemoveContainer" containerID="4149f24d3cc318e962f2c40a15f315175b0a27b626727cfc1f3571cd2f6f0fad"
Oct 30 23:44:16.127874 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:16.127857    5078 scope.go:122] "RemoveContainer" containerID="a9a1da734944bbd0a55c88375283d87a7fc4634a4df94df2bd704ab535c62b27"
Oct 30 23:44:16.138896 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:16.138871    5078 scope.go:122] "RemoveContainer" containerID="4149f24d3cc318e962f2c40a15f315175b0a27b626727cfc1f3571cd2f6f0fad"
Oct 30 23:44:16.139127 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:44:16.139104    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4149f24d3cc318e962f2c40a15f315175b0a27b626727cfc1f3571cd2f6f0fad\": container with ID starting with 4149f24d3cc318e962f2c40a15f315175b0a27b626727cfc1f3571cd2f6f0fad not found: ID does not exist" containerID="4149f24d3cc318e962f2c40a15f315175b0a27b626727cfc1f3571cd2f6f0fad"
Oct 30 23:44:16.139234 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:16.139138    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4149f24d3cc318e962f2c40a15f315175b0a27b626727cfc1f3571cd2f6f0fad"} err="failed to get container status \"4149f24d3cc318e962f2c40a15f315175b0a27b626727cfc1f3571cd2f6f0fad\": rpc error: code = NotFound desc = could not find container \"4149f24d3cc318e962f2c40a15f315175b0a27b626727cfc1f3571cd2f6f0fad\": container with ID starting with 4149f24d3cc318e962f2c40a15f315175b0a27b626727cfc1f3571cd2f6f0fad not found: ID does not exist"
Oct 30 23:44:16.139234 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:16.139163    5078 scope.go:122] "RemoveContainer" containerID="a9a1da734944bbd0a55c88375283d87a7fc4634a4df94df2bd704ab535c62b27"
Oct 30 23:44:16.139397 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:44:16.139372    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9a1da734944bbd0a55c88375283d87a7fc4634a4df94df2bd704ab535c62b27\": container with ID starting with a9a1da734944bbd0a55c88375283d87a7fc4634a4df94df2bd704ab535c62b27 not found: ID does not exist" containerID="a9a1da734944bbd0a55c88375283d87a7fc4634a4df94df2bd704ab535c62b27"
Oct 30 23:44:16.139468 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:16.139406    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9a1da734944bbd0a55c88375283d87a7fc4634a4df94df2bd704ab535c62b27"} err="failed to get container status \"a9a1da734944bbd0a55c88375283d87a7fc4634a4df94df2bd704ab535c62b27\": rpc error: code = NotFound desc = could not find container \"a9a1da734944bbd0a55c88375283d87a7fc4634a4df94df2bd704ab535c62b27\": container with ID starting with a9a1da734944bbd0a55c88375283d87a7fc4634a4df94df2bd704ab535c62b27 not found: ID does not exist"
Oct 30 23:44:16.286291 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:16.286259    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"]
Oct 30 23:44:16.335583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:16.335555    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-196ff7-openshift-default-86d7f48d85-7z49r"]
Oct 30 23:44:16.862142 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:16.862112    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h"]
Oct 30 23:44:16.865291 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:16.865267    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7d1536-qpw8h"]
Oct 30 23:44:17.600132 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:44:17.600103    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:44:17.822385 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.822360    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:44:17.841735 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.841678    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-kube-api-access-68c5r\" (UniqueName: \"kubernetes.io/projected/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-kube-api-access-68c5r\") pod \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") "
Oct 30 23:44:17.841849 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.841777    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-workload-certs\") pod \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") "
Oct 30 23:44:17.841905 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.841849    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-workload-socket\") pod \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") "
Oct 30 23:44:17.841905 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.841881    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-data\") pod \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") "
Oct 30 23:44:17.842016 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.841970    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-envoy\") pod \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") "
Oct 30 23:44:17.842081 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.842044    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istiod-ca-cert\") pod \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") "
Oct 30 23:44:17.842141 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.842063    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-workload-certs" pod "4cf62568-28ad-43c6-a0ab-f1680ef1a72c" (UID: "4cf62568-28ad-43c6-a0ab-f1680ef1a72c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:17.842141 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.842108    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-credential-socket\") pod \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") "
Oct 30 23:44:17.842261 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.842133    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-workload-socket" pod "4cf62568-28ad-43c6-a0ab-f1680ef1a72c" (UID: "4cf62568-28ad-43c6-a0ab-f1680ef1a72c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:17.842261 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.842172    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-podinfo\") pod \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") "
Oct 30 23:44:17.842261 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.842200    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-token\" (UniqueName: \"kubernetes.io/projected/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-token\") pod \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\" (UID: \"4cf62568-28ad-43c6-a0ab-f1680ef1a72c\") "
Oct 30 23:44:17.842410 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.842244    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-data" pod "4cf62568-28ad-43c6-a0ab-f1680ef1a72c" (UID: "4cf62568-28ad-43c6-a0ab-f1680ef1a72c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:17.842471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.842403    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-credential-socket" pod "4cf62568-28ad-43c6-a0ab-f1680ef1a72c" (UID: "4cf62568-28ad-43c6-a0ab-f1680ef1a72c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:17.842471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.842412    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istiod-ca-cert" pod "4cf62568-28ad-43c6-a0ab-f1680ef1a72c" (UID: "4cf62568-28ad-43c6-a0ab-f1680ef1a72c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:44:17.843075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.843051    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:17.843184 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.843081    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:17.843184 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.843098    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:17.843184 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.843112    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:17.843184 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.843127    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:17.844471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.844445    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-podinfo" pod "4cf62568-28ad-43c6-a0ab-f1680ef1a72c" (UID: "4cf62568-28ad-43c6-a0ab-f1680ef1a72c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:44:17.844581 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.844561    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-token" pod "4cf62568-28ad-43c6-a0ab-f1680ef1a72c" (UID: "4cf62568-28ad-43c6-a0ab-f1680ef1a72c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:17.844699 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.844625    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-envoy" pod "4cf62568-28ad-43c6-a0ab-f1680ef1a72c" (UID: "4cf62568-28ad-43c6-a0ab-f1680ef1a72c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:17.844924 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.844910    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-kube-api-access-68c5r" pod "4cf62568-28ad-43c6-a0ab-f1680ef1a72c" (UID: "4cf62568-28ad-43c6-a0ab-f1680ef1a72c"). InnerVolumeSpecName "kube-api-access-68c5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:17.944145 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.944111    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68c5r\" (UniqueName: \"kubernetes.io/projected/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-kube-api-access-68c5r\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:17.944145 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.944141    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:17.944525 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.944155    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:17.944525 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:17.944167    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cf62568-28ad-43c6-a0ab-f1680ef1a72c-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:18.138688 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:18.138589    5078 generic.go:356] "Generic (PLEG): container finished" podID="4cf62568-28ad-43c6-a0ab-f1680ef1a72c" containerID="7f56c1761fcc608527aed38a6bc595c602d514bce08473de1db7effb2cd8c077" exitCode=0
Oct 30 23:44:18.138688 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:18.138679    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"
Oct 30 23:44:18.138933 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:18.138673    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv" event={"ID":"4cf62568-28ad-43c6-a0ab-f1680ef1a72c","Type":"ContainerDied","Data":"7f56c1761fcc608527aed38a6bc595c602d514bce08473de1db7effb2cd8c077"}
Oct 30 23:44:18.138933 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:18.138772    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv" event={"ID":"4cf62568-28ad-43c6-a0ab-f1680ef1a72c","Type":"ContainerDied","Data":"e40bbeb65759e87ddabee11c1a5d9217c96cba624209364fa83daaf167d6acfa"}
Oct 30 23:44:18.138933 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:18.138806    5078 scope.go:122] "RemoveContainer" containerID="7f56c1761fcc608527aed38a6bc595c602d514bce08473de1db7effb2cd8c077"
Oct 30 23:44:18.152302 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:18.152283    5078 scope.go:122] "RemoveContainer" containerID="dadf087ab6b027c3f6a6f2cc1e9e2daf54de156826205181c8f938b27e1afc16"
Oct 30 23:44:18.161438 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:18.161421    5078 scope.go:122] "RemoveContainer" containerID="7f56c1761fcc608527aed38a6bc595c602d514bce08473de1db7effb2cd8c077"
Oct 30 23:44:18.161737 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:44:18.161718    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f56c1761fcc608527aed38a6bc595c602d514bce08473de1db7effb2cd8c077\": container with ID starting with 7f56c1761fcc608527aed38a6bc595c602d514bce08473de1db7effb2cd8c077 not found: ID does not exist" containerID="7f56c1761fcc608527aed38a6bc595c602d514bce08473de1db7effb2cd8c077"
Oct 30 23:44:18.161814 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:18.161743    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f56c1761fcc608527aed38a6bc595c602d514bce08473de1db7effb2cd8c077"} err="failed to get container status \"7f56c1761fcc608527aed38a6bc595c602d514bce08473de1db7effb2cd8c077\": rpc error: code = NotFound desc = could not find container \"7f56c1761fcc608527aed38a6bc595c602d514bce08473de1db7effb2cd8c077\": container with ID starting with 7f56c1761fcc608527aed38a6bc595c602d514bce08473de1db7effb2cd8c077 not found: ID does not exist"
Oct 30 23:44:18.161814 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:18.161762    5078 scope.go:122] "RemoveContainer" containerID="dadf087ab6b027c3f6a6f2cc1e9e2daf54de156826205181c8f938b27e1afc16"
Oct 30 23:44:18.162012 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:44:18.161995    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dadf087ab6b027c3f6a6f2cc1e9e2daf54de156826205181c8f938b27e1afc16\": container with ID starting with dadf087ab6b027c3f6a6f2cc1e9e2daf54de156826205181c8f938b27e1afc16 not found: ID does not exist" containerID="dadf087ab6b027c3f6a6f2cc1e9e2daf54de156826205181c8f938b27e1afc16"
Oct 30 23:44:18.162054 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:18.162018    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dadf087ab6b027c3f6a6f2cc1e9e2daf54de156826205181c8f938b27e1afc16"} err="failed to get container status \"dadf087ab6b027c3f6a6f2cc1e9e2daf54de156826205181c8f938b27e1afc16\": rpc error: code = NotFound desc = could not find container \"dadf087ab6b027c3f6a6f2cc1e9e2daf54de156826205181c8f938b27e1afc16\": container with ID starting with dadf087ab6b027c3f6a6f2cc1e9e2daf54de156826205181c8f938b27e1afc16 not found: ID does not exist"
Oct 30 23:44:18.298799 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:18.298761    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c1392b-ce18-4918-b612-1f1424dafd7e" path="/var/lib/kubelet/pods/b9c1392b-ce18-4918-b612-1f1424dafd7e/volumes"
Oct 30 23:44:18.299343 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:18.299321    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc173733-80d8-4793-b1a1-723844b99673" path="/var/lib/kubelet/pods/fc173733-80d8-4793-b1a1-723844b99673/volumes"
Oct 30 23:44:18.313654 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:18.313632    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"]
Oct 30 23:44:18.317423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:18.317403    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-7a42d8-openshift-default-85c4ccf595-z65cv"]
Oct 30 23:44:20.299982 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:20.299950    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cf62568-28ad-43c6-a0ab-f1680ef1a72c" path="/var/lib/kubelet/pods/4cf62568-28ad-43c6-a0ab-f1680ef1a72c/volumes"
Oct 30 23:44:22.921649 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:22.920982    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz"]
Oct 30 23:44:22.974087 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:22.974047    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk"]
Oct 30 23:44:23.528730 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:23.528656    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8"]
Oct 30 23:44:23.542045 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:23.542007    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-0a7530-cxvh8"]
Oct 30 23:44:24.358274 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:24.358239    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f46e2510-bd1d-41c2-9ca0-e785f5bb0133" path="/var/lib/kubelet/pods/f46e2510-bd1d-41c2-9ca0-e785f5bb0133/volumes"
Oct 30 23:44:25.085303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:25.085259    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"]
Oct 30 23:44:25.085808 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:25.085775    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26" podUID="4539c4b3-4808-4706-b794-82d7a2fa6034" containerName="istio-proxy" containerID="cri-o://669052311b2e8ecc352f4cd6f919b78b81b3eb8773b5b1ea17d77a8150844dac" gracePeriod=30
Oct 30 23:44:26.196264 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:26.196233    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv"]
Oct 30 23:44:26.198893 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:26.198871    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ba13e7-8p9gv"]
Oct 30 23:44:26.298253 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:26.298221    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99e66e43-6a0c-4b97-be01-a65a5ec5fd70" path="/var/lib/kubelet/pods/99e66e43-6a0c-4b97-be01-a65a5ec5fd70/volumes"
Oct 30 23:44:27.210977 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.210941    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr"]
Oct 30 23:44:27.211538 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.211520    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4cf62568-28ad-43c6-a0ab-f1680ef1a72c" containerName="istio-proxy"
Oct 30 23:44:27.211612 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.211547    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf62568-28ad-43c6-a0ab-f1680ef1a72c" containerName="istio-proxy"
Oct 30 23:44:27.211612 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.211563    5078 cpu_manager.go:414] "containerMap: removing container" podUID="66fc5643-d66c-4cb5-98c7-15af84bf7167" containerName="revoke-keys"
Oct 30 23:44:27.211612 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.211568    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66fc5643-d66c-4cb5-98c7-15af84bf7167" containerName="revoke-keys"
Oct 30 23:44:27.211612 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.211591    5078 cpu_manager.go:414] "containerMap: removing container" podUID="fc173733-80d8-4793-b1a1-723844b99673" containerName="istio-proxy"
Oct 30 23:44:27.211764 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.211612    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc173733-80d8-4793-b1a1-723844b99673" containerName="istio-proxy"
Oct 30 23:44:27.211764 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.211640    5078 cpu_manager.go:414] "containerMap: removing container" podUID="fc173733-80d8-4793-b1a1-723844b99673" containerName="istio-proxy"
Oct 30 23:44:27.211764 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.211647    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc173733-80d8-4793-b1a1-723844b99673" containerName="istio-proxy"
Oct 30 23:44:27.211764 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.211754    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf62568-28ad-43c6-a0ab-f1680ef1a72c" containerName="istio-proxy"
Oct 30 23:44:27.211980 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.211768    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="66fc5643-d66c-4cb5-98c7-15af84bf7167" containerName="revoke-keys"
Oct 30 23:44:27.211980 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.211778    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc173733-80d8-4793-b1a1-723844b99673" containerName="istio-proxy"
Oct 30 23:44:27.211980 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.211786    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc173733-80d8-4793-b1a1-723844b99673" containerName="istio-proxy"
Oct 30 23:44:27.217001 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.216977    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr"
Oct 30 23:44:27.272265 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.272236    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr"]
Oct 30 23:44:27.318034 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.318002    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:44:27.318034 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.318021    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:44:27.318034 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.318023    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:44:27.318712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.318699    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:44:27.393985 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.393948    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kn2q\" (UniqueName: \"kubernetes.io/projected/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-kube-api-access-4kn2q\") pod \"maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr\" (UID: \"7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr"
Oct 30 23:44:27.394190 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.394017    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr\" (UID: \"7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr"
Oct 30 23:44:27.394277 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.394258    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr\" (UID: \"7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr"
Oct 30 23:44:27.496116 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.496032    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr\" (UID: \"7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr"
Oct 30 23:44:27.496303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.496134    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kn2q\" (UniqueName: \"kubernetes.io/projected/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-kube-api-access-4kn2q\") pod \"maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr\" (UID: \"7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr"
Oct 30 23:44:27.496303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.496174    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr\" (UID: \"7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr"
Oct 30 23:44:27.496781 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.496754    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr\" (UID: \"7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr"
Oct 30 23:44:27.498581 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.498562    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr\" (UID: \"7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr"
Oct 30 23:44:27.553354 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.553327    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kn2q\" (UniqueName: \"kubernetes.io/projected/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-kube-api-access-4kn2q\") pod \"maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr\" (UID: \"7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr"
Oct 30 23:44:27.830865 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.830830    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr"
Oct 30 23:44:27.925866 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.925227    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz" podUID="b7506656-8e86-4ecc-a0d7-41d6c87ddb67" containerName="payload-processing" containerID="cri-o://28859c60576178ab9cc24ad010bb88ca64f4141257a06a740128062d662645fd" gracePeriod=25
Oct 30 23:44:27.974458 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.974417    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk" podUID="252b60c8-71fc-470b-91d3-7f6586510c69" containerName="payload-pre-processing" containerID="cri-o://5e164c8c5e29b867e04c28493e87a8542dcad3374edb22ac1f8a1d575a5188e4" gracePeriod=25
Oct 30 23:44:27.989358 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.989332    5078 patch_prober.go:29] interesting pod/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:44:27.989471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:27.989386    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26" podUID="4539c4b3-4808-4706-b794-82d7a2fa6034" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:44:28.207956 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.207581    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr"]
Oct 30 23:44:28.234569 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.234484    5078 generic.go:356] "Generic (PLEG): container finished" podID="252b60c8-71fc-470b-91d3-7f6586510c69" containerID="5e164c8c5e29b867e04c28493e87a8542dcad3374edb22ac1f8a1d575a5188e4" exitCode=0
Oct 30 23:44:28.235017 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.234669    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk" event={"ID":"252b60c8-71fc-470b-91d3-7f6586510c69","Type":"ContainerDied","Data":"5e164c8c5e29b867e04c28493e87a8542dcad3374edb22ac1f8a1d575a5188e4"}
Oct 30 23:44:28.237010 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.236960    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr" event={"ID":"7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501","Type":"ContainerStarted","Data":"7c4618948338a12455089ec77f1893361cc2ae8dc34a9b186ff58829cdf10889"}
Oct 30 23:44:28.239081 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.238923    5078 generic.go:356] "Generic (PLEG): container finished" podID="b7506656-8e86-4ecc-a0d7-41d6c87ddb67" containerID="28859c60576178ab9cc24ad010bb88ca64f4141257a06a740128062d662645fd" exitCode=0
Oct 30 23:44:28.239081 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.238974    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz" event={"ID":"b7506656-8e86-4ecc-a0d7-41d6c87ddb67","Type":"ContainerDied","Data":"28859c60576178ab9cc24ad010bb88ca64f4141257a06a740128062d662645fd"}
Oct 30 23:44:28.241268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.241243    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz"
Oct 30 23:44:28.294296 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.294270    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk"
Oct 30 23:44:28.409245 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.409219    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/252b60c8-71fc-470b-91d3-7f6586510c69-kube-api-access-lmlw6\" (UniqueName: \"kubernetes.io/projected/252b60c8-71fc-470b-91d3-7f6586510c69-kube-api-access-lmlw6\") pod \"252b60c8-71fc-470b-91d3-7f6586510c69\" (UID: \"252b60c8-71fc-470b-91d3-7f6586510c69\") "
Oct 30 23:44:28.409447 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.409305    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/252b60c8-71fc-470b-91d3-7f6586510c69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252b60c8-71fc-470b-91d3-7f6586510c69-plugins-config-volume\") pod \"252b60c8-71fc-470b-91d3-7f6586510c69\" (UID: \"252b60c8-71fc-470b-91d3-7f6586510c69\") "
Oct 30 23:44:28.409447 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.409350    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7506656-8e86-4ecc-a0d7-41d6c87ddb67-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7506656-8e86-4ecc-a0d7-41d6c87ddb67-plugins-config-volume\") pod \"b7506656-8e86-4ecc-a0d7-41d6c87ddb67\" (UID: \"b7506656-8e86-4ecc-a0d7-41d6c87ddb67\") "
Oct 30 23:44:28.409583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.409484    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7506656-8e86-4ecc-a0d7-41d6c87ddb67-kube-api-access-2865t\" (UniqueName: \"kubernetes.io/projected/b7506656-8e86-4ecc-a0d7-41d6c87ddb67-kube-api-access-2865t\") pod \"b7506656-8e86-4ecc-a0d7-41d6c87ddb67\" (UID: \"b7506656-8e86-4ecc-a0d7-41d6c87ddb67\") "
Oct 30 23:44:28.409757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.409727    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252b60c8-71fc-470b-91d3-7f6586510c69-plugins-config-volume" pod "252b60c8-71fc-470b-91d3-7f6586510c69" (UID: "252b60c8-71fc-470b-91d3-7f6586510c69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:44:28.409829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.409750    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7506656-8e86-4ecc-a0d7-41d6c87ddb67-plugins-config-volume" pod "b7506656-8e86-4ecc-a0d7-41d6c87ddb67" (UID: "b7506656-8e86-4ecc-a0d7-41d6c87ddb67"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:44:28.410175 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.410157    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/252b60c8-71fc-470b-91d3-7f6586510c69-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:28.410222 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.410181    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7506656-8e86-4ecc-a0d7-41d6c87ddb67-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:28.411818 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.411801    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7506656-8e86-4ecc-a0d7-41d6c87ddb67-kube-api-access-2865t" pod "b7506656-8e86-4ecc-a0d7-41d6c87ddb67" (UID: "b7506656-8e86-4ecc-a0d7-41d6c87ddb67"). InnerVolumeSpecName "kube-api-access-2865t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:28.411874 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.411822    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252b60c8-71fc-470b-91d3-7f6586510c69-kube-api-access-lmlw6" pod "252b60c8-71fc-470b-91d3-7f6586510c69" (UID: "252b60c8-71fc-470b-91d3-7f6586510c69"). InnerVolumeSpecName "kube-api-access-lmlw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:28.512074 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.511942    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmlw6\" (UniqueName: \"kubernetes.io/projected/252b60c8-71fc-470b-91d3-7f6586510c69-kube-api-access-lmlw6\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:28.512074 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:28.512051    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2865t\" (UniqueName: \"kubernetes.io/projected/b7506656-8e86-4ecc-a0d7-41d6c87ddb67-kube-api-access-2865t\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:29.247650 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:29.247610    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz" event={"ID":"b7506656-8e86-4ecc-a0d7-41d6c87ddb67","Type":"ContainerDied","Data":"383479a2f680110a2ba81850f14693e9936d6af3485f148925e769945172883b"}
Oct 30 23:44:29.248087 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:29.247629    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz"
Oct 30 23:44:29.248087 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:29.247681    5078 scope.go:122] "RemoveContainer" containerID="28859c60576178ab9cc24ad010bb88ca64f4141257a06a740128062d662645fd"
Oct 30 23:44:29.249560 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:29.249542    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk"
Oct 30 23:44:29.249671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:29.249559    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk" event={"ID":"252b60c8-71fc-470b-91d3-7f6586510c69","Type":"ContainerDied","Data":"11918711f60f625f677d2b8fe8825fcb6d6ddc40c0596e85f26d57f1e417386f"}
Oct 30 23:44:29.251458 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:29.251430    5078 generic.go:356] "Generic (PLEG): container finished" podID="7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501" containerID="2a3229f2f839a4452a217f153dd1e28fc700f68dfcb59fe36853225fa85a6d0f" exitCode=0
Oct 30 23:44:29.251589 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:29.251507    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr" event={"ID":"7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501","Type":"ContainerDied","Data":"2a3229f2f839a4452a217f153dd1e28fc700f68dfcb59fe36853225fa85a6d0f"}
Oct 30 23:44:29.258956 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:29.258941    5078 scope.go:122] "RemoveContainer" containerID="5e164c8c5e29b867e04c28493e87a8542dcad3374edb22ac1f8a1d575a5188e4"
Oct 30 23:44:29.465013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:29.464825    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz"]
Oct 30 23:44:29.465013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:29.464880    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-59d267-858d8977-hvjbz"]
Oct 30 23:44:29.567178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:29.567139    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk"]
Oct 30 23:44:29.573888 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:29.573858    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-59d267-5d588f69f5-qbnsk"]
Oct 30 23:44:30.105306 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.105273    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"]
Oct 30 23:44:30.105581 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.105555    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6" podUID="ae2a127b-3a86-45d1-8e4a-10557a96a711" containerName="istio-proxy" containerID="cri-o://21d276b2ca00ef3304604872f3206f84b51c0b3d12e0be882544e5757d26bcc9" gracePeriod=30
Oct 30 23:44:30.382433 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.382397    5078 generic.go:356] "Generic (PLEG): container finished" podID="4539c4b3-4808-4706-b794-82d7a2fa6034" containerID="669052311b2e8ecc352f4cd6f919b78b81b3eb8773b5b1ea17d77a8150844dac" exitCode=0
Oct 30 23:44:30.386783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.386748    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="252b60c8-71fc-470b-91d3-7f6586510c69" path="/var/lib/kubelet/pods/252b60c8-71fc-470b-91d3-7f6586510c69/volumes"
Oct 30 23:44:30.387723 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.387697    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7506656-8e86-4ecc-a0d7-41d6c87ddb67" path="/var/lib/kubelet/pods/b7506656-8e86-4ecc-a0d7-41d6c87ddb67/volumes"
Oct 30 23:44:30.388201 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.388178    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26" event={"ID":"4539c4b3-4808-4706-b794-82d7a2fa6034","Type":"ContainerDied","Data":"669052311b2e8ecc352f4cd6f919b78b81b3eb8773b5b1ea17d77a8150844dac"}
Oct 30 23:44:30.388274 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.388263    5078 scope.go:122] "RemoveContainer" containerID="370b58a9d4d9006ab5ca5ea920af855449df7322bc79d00cea53277ca522310c"
Oct 30 23:44:30.495742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.495721    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:44:30.558175 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.558148    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-workload-certs\") pod \"4539c4b3-4808-4706-b794-82d7a2fa6034\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") "
Oct 30 23:44:30.558315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.558249    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4539c4b3-4808-4706-b794-82d7a2fa6034-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4539c4b3-4808-4706-b794-82d7a2fa6034-istiod-ca-cert\") pod \"4539c4b3-4808-4706-b794-82d7a2fa6034\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") "
Oct 30 23:44:30.558392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.558319    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-credential-socket\") pod \"4539c4b3-4808-4706-b794-82d7a2fa6034\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") "
Oct 30 23:44:30.558392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.558364    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-workload-certs" pod "4539c4b3-4808-4706-b794-82d7a2fa6034" (UID: "4539c4b3-4808-4706-b794-82d7a2fa6034"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:30.558392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.558371    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-podinfo\") pod \"4539c4b3-4808-4706-b794-82d7a2fa6034\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") "
Oct 30 23:44:30.558559 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.558427    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-token\" (UniqueName: \"kubernetes.io/projected/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-token\") pod \"4539c4b3-4808-4706-b794-82d7a2fa6034\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") "
Oct 30 23:44:30.558559 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.558517    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-workload-socket\") pod \"4539c4b3-4808-4706-b794-82d7a2fa6034\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") "
Oct 30 23:44:30.558723 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.558691    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-envoy\") pod \"4539c4b3-4808-4706-b794-82d7a2fa6034\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") "
Oct 30 23:44:30.558825 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.558742    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4539c4b3-4808-4706-b794-82d7a2fa6034-istiod-ca-cert" pod "4539c4b3-4808-4706-b794-82d7a2fa6034" (UID: "4539c4b3-4808-4706-b794-82d7a2fa6034"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:44:30.558825 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.558796    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4539c4b3-4808-4706-b794-82d7a2fa6034-kube-api-access-rnl24\" (UniqueName: \"kubernetes.io/projected/4539c4b3-4808-4706-b794-82d7a2fa6034-kube-api-access-rnl24\") pod \"4539c4b3-4808-4706-b794-82d7a2fa6034\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") "
Oct 30 23:44:30.559009 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.558850    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-data\") pod \"4539c4b3-4808-4706-b794-82d7a2fa6034\" (UID: \"4539c4b3-4808-4706-b794-82d7a2fa6034\") "
Oct 30 23:44:30.559009 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.558989    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-workload-socket" pod "4539c4b3-4808-4706-b794-82d7a2fa6034" (UID: "4539c4b3-4808-4706-b794-82d7a2fa6034"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:30.559299 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.559173    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-credential-socket" pod "4539c4b3-4808-4706-b794-82d7a2fa6034" (UID: "4539c4b3-4808-4706-b794-82d7a2fa6034"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:30.559372 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.559351    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-data" pod "4539c4b3-4808-4706-b794-82d7a2fa6034" (UID: "4539c4b3-4808-4706-b794-82d7a2fa6034"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:30.559480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.559457    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:30.559553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.559486    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:30.559553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.559500    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4539c4b3-4808-4706-b794-82d7a2fa6034-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:30.559553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.559512    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:30.562073 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.562029    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-token" pod "4539c4b3-4808-4706-b794-82d7a2fa6034" (UID: "4539c4b3-4808-4706-b794-82d7a2fa6034"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:30.562073 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.562039    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4539c4b3-4808-4706-b794-82d7a2fa6034-kube-api-access-rnl24" pod "4539c4b3-4808-4706-b794-82d7a2fa6034" (UID: "4539c4b3-4808-4706-b794-82d7a2fa6034"). InnerVolumeSpecName "kube-api-access-rnl24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:30.562235 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.562092    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-envoy" pod "4539c4b3-4808-4706-b794-82d7a2fa6034" (UID: "4539c4b3-4808-4706-b794-82d7a2fa6034"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:30.562235 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.562202    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-podinfo" pod "4539c4b3-4808-4706-b794-82d7a2fa6034" (UID: "4539c4b3-4808-4706-b794-82d7a2fa6034"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:44:30.585848 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.585824    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr"
Oct 30 23:44:30.660851 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.660773    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-maas-api-xks-ca\") pod \"7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501\" (UID: \"7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501\") "
Oct 30 23:44:30.661008 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.660862    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-maas-api-openshift-ca\") pod \"7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501\" (UID: \"7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501\") "
Oct 30 23:44:30.661008 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.660922    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-kube-api-access-4kn2q\" (UniqueName: \"kubernetes.io/projected/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-kube-api-access-4kn2q\") pod \"7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501\" (UID: \"7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501\") "
Oct 30 23:44:30.661252 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.661230    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-maas-api-openshift-ca" pod "7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501" (UID: "7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:44:30.661409 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.661386    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:30.661409 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.661411    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:30.661568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.661424    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:30.661568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.661437    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:30.661568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.661451    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4539c4b3-4808-4706-b794-82d7a2fa6034-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:30.661568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.661464    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnl24\" (UniqueName: \"kubernetes.io/projected/4539c4b3-4808-4706-b794-82d7a2fa6034-kube-api-access-rnl24\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:30.663185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.663163    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-kube-api-access-4kn2q" pod "7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501" (UID: "7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501"). InnerVolumeSpecName "kube-api-access-4kn2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:30.663245 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.663164    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-maas-api-xks-ca" pod "7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501" (UID: "7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:44:30.762445 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.762418    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:30.762445 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:30.762443    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kn2q\" (UniqueName: \"kubernetes.io/projected/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501-kube-api-access-4kn2q\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:31.330460 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:31.330428    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5"]
Oct 30 23:44:31.333716 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:31.333693    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-59d267-nscs5"]
Oct 30 23:44:31.390463 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:31.390437    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"
Oct 30 23:44:31.390919 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:31.390462    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26" event={"ID":"4539c4b3-4808-4706-b794-82d7a2fa6034","Type":"ContainerDied","Data":"f96669a80713b6f6c4e672603f958abd5ff03dbbd7e801f76bcaabfb9d653b64"}
Oct 30 23:44:31.390919 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:31.390520    5078 scope.go:122] "RemoveContainer" containerID="669052311b2e8ecc352f4cd6f919b78b81b3eb8773b5b1ea17d77a8150844dac"
Oct 30 23:44:31.392067 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:31.392043    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr" event={"ID":"7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501","Type":"ContainerDied","Data":"7c4618948338a12455089ec77f1893361cc2ae8dc34a9b186ff58829cdf10889"}
Oct 30 23:44:31.392183 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:31.392078    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr"
Oct 30 23:44:31.392183 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:31.392085    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c4618948338a12455089ec77f1893361cc2ae8dc34a9b186ff58829cdf10889"
Oct 30 23:44:31.555185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:31.555157    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"]
Oct 30 23:44:31.559301 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:31.559274    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7d1536-openshift-default-74d4788bf9-t6x26"]
Oct 30 23:44:31.671185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:31.671153    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"]
Oct 30 23:44:31.675800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:31.675775    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"]
Oct 30 23:44:31.683643 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:31.683621    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"]
Oct 30 23:44:32.298702 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:32.298670    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4539c4b3-4808-4706-b794-82d7a2fa6034" path="/var/lib/kubelet/pods/4539c4b3-4808-4706-b794-82d7a2fa6034/volumes"
Oct 30 23:44:32.299132 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:32.299118    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66fc5643-d66c-4cb5-98c7-15af84bf7167" path="/var/lib/kubelet/pods/66fc5643-d66c-4cb5-98c7-15af84bf7167/volumes"
Oct 30 23:44:35.053029 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.052945    5078 patch_prober.go:29] interesting pod/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:44:35.053029 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.053007    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6" podUID="ae2a127b-3a86-45d1-8e4a-10557a96a711" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:44:35.197401 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.197368    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"]
Oct 30 23:44:35.197961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.197946    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4cf62568-28ad-43c6-a0ab-f1680ef1a72c" containerName="istio-proxy"
Oct 30 23:44:35.198018 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.197962    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf62568-28ad-43c6-a0ab-f1680ef1a72c" containerName="istio-proxy"
Oct 30 23:44:35.198018 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.197989    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4539c4b3-4808-4706-b794-82d7a2fa6034" containerName="istio-proxy"
Oct 30 23:44:35.198018 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.197997    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4539c4b3-4808-4706-b794-82d7a2fa6034" containerName="istio-proxy"
Oct 30 23:44:35.198018 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.198011    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4539c4b3-4808-4706-b794-82d7a2fa6034" containerName="istio-proxy"
Oct 30 23:44:35.198018 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.198016    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4539c4b3-4808-4706-b794-82d7a2fa6034" containerName="istio-proxy"
Oct 30 23:44:35.198169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.198027    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b7506656-8e86-4ecc-a0d7-41d6c87ddb67" containerName="payload-processing"
Oct 30 23:44:35.198169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.198031    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7506656-8e86-4ecc-a0d7-41d6c87ddb67" containerName="payload-processing"
Oct 30 23:44:35.198169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.198039    5078 cpu_manager.go:414] "containerMap: removing container" podUID="252b60c8-71fc-470b-91d3-7f6586510c69" containerName="payload-pre-processing"
Oct 30 23:44:35.198169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.198044    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="252b60c8-71fc-470b-91d3-7f6586510c69" containerName="payload-pre-processing"
Oct 30 23:44:35.198169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.198054    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501" containerName="revoke-keys"
Oct 30 23:44:35.198169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.198059    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501" containerName="revoke-keys"
Oct 30 23:44:35.198169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.198130    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7506656-8e86-4ecc-a0d7-41d6c87ddb67" containerName="payload-processing"
Oct 30 23:44:35.198169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.198139    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="252b60c8-71fc-470b-91d3-7f6586510c69" containerName="payload-pre-processing"
Oct 30 23:44:35.198169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.198145    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501" containerName="revoke-keys"
Oct 30 23:44:35.198169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.198152    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf62568-28ad-43c6-a0ab-f1680ef1a72c" containerName="istio-proxy"
Oct 30 23:44:35.198169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.198161    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4539c4b3-4808-4706-b794-82d7a2fa6034" containerName="istio-proxy"
Oct 30 23:44:35.198169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.198169    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4539c4b3-4808-4706-b794-82d7a2fa6034" containerName="istio-proxy"
Oct 30 23:44:35.201586 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.201569    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.248622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.248506    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-istio-envoy\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.248622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.248617    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412494b5-417a-488e-9ce4-9771d6458325-istio-podinfo\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.248804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.248751    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ms5b\" (UniqueName: \"kubernetes.io/projected/412494b5-417a-488e-9ce4-9771d6458325-kube-api-access-6ms5b\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.248804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.248778    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412494b5-417a-488e-9ce4-9771d6458325-istiod-ca-cert\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.248922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.248803    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/412494b5-417a-488e-9ce4-9771d6458325-istio-token\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.248922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.248854    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-istio-data\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.248922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.248882    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-workload-socket\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.249093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.248962    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-credential-socket\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.249093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.249014    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-workload-certs\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.260322 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.258958    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-71a68a-openshift-default-dockercfg-hdm8z\""
Oct 30 23:44:35.399678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.350373    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412494b5-417a-488e-9ce4-9771d6458325-istio-podinfo\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.399678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.350510    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ms5b\" (UniqueName: \"kubernetes.io/projected/412494b5-417a-488e-9ce4-9771d6458325-kube-api-access-6ms5b\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.399678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.350539    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412494b5-417a-488e-9ce4-9771d6458325-istiod-ca-cert\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.399678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.350565    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/412494b5-417a-488e-9ce4-9771d6458325-istio-token\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.399678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.359918    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-istio-data\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.399678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.359967    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-workload-socket\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.399678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.360108    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-credential-socket\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.399678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.360177    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-workload-certs\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.399678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.360286    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-istio-envoy\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.399678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.364103    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-workload-socket\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.399678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.364335    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-istio-data\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.399678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.364952    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412494b5-417a-488e-9ce4-9771d6458325-istiod-ca-cert\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.399678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.365185    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-credential-socket\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.399678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.365407    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-workload-certs\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.423315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.407055    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"]
Oct 30 23:44:35.507153 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.507117    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/412494b5-417a-488e-9ce4-9771d6458325-istio-token\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.510507 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.510336    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412494b5-417a-488e-9ce4-9771d6458325-istio-podinfo\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.510507 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.510450    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-istio-envoy\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.514245 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.514192    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ms5b\" (UniqueName: \"kubernetes.io/projected/412494b5-417a-488e-9ce4-9771d6458325-kube-api-access-6ms5b\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.518680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.518654    5078 generic.go:356] "Generic (PLEG): container finished" podID="ae2a127b-3a86-45d1-8e4a-10557a96a711" containerID="21d276b2ca00ef3304604872f3206f84b51c0b3d12e0be882544e5757d26bcc9" exitCode=0
Oct 30 23:44:35.518840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.518826    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6" event={"ID":"ae2a127b-3a86-45d1-8e4a-10557a96a711","Type":"ContainerDied","Data":"21d276b2ca00ef3304604872f3206f84b51c0b3d12e0be882544e5757d26bcc9"}
Oct 30 23:44:35.518884 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.518872    5078 scope.go:122] "RemoveContainer" containerID="aa59d354ec5d91a535b2c4bafd7d0a0c257a72098eb79864d3f9d74a7fda6edd"
Oct 30 23:44:35.530973 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.530843    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:35.710086 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.710010    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:44:35.880777 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.880751    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2a127b-3a86-45d1-8e4a-10557a96a711-kube-api-access-w5276\" (UniqueName: \"kubernetes.io/projected/ae2a127b-3a86-45d1-8e4a-10557a96a711-kube-api-access-w5276\") pod \"ae2a127b-3a86-45d1-8e4a-10557a96a711\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") "
Oct 30 23:44:35.880951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.880856    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2a127b-3a86-45d1-8e4a-10557a96a711-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2a127b-3a86-45d1-8e4a-10557a96a711-istiod-ca-cert\") pod \"ae2a127b-3a86-45d1-8e4a-10557a96a711\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") "
Oct 30 23:44:35.880951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.880902    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-token\" (UniqueName: \"kubernetes.io/projected/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-token\") pod \"ae2a127b-3a86-45d1-8e4a-10557a96a711\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") "
Oct 30 23:44:35.880951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.880924    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-credential-socket\") pod \"ae2a127b-3a86-45d1-8e4a-10557a96a711\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") "
Oct 30 23:44:35.881120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.880977    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-workload-socket\") pod \"ae2a127b-3a86-45d1-8e4a-10557a96a711\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") "
Oct 30 23:44:35.881120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.880996    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-data\") pod \"ae2a127b-3a86-45d1-8e4a-10557a96a711\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") "
Oct 30 23:44:35.881120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.881017    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-workload-certs\") pod \"ae2a127b-3a86-45d1-8e4a-10557a96a711\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") "
Oct 30 23:44:35.881120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.881101    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-envoy\") pod \"ae2a127b-3a86-45d1-8e4a-10557a96a711\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") "
Oct 30 23:44:35.881364 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.881228    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-podinfo\") pod \"ae2a127b-3a86-45d1-8e4a-10557a96a711\" (UID: \"ae2a127b-3a86-45d1-8e4a-10557a96a711\") "
Oct 30 23:44:35.881364 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.881233    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-credential-socket" pod "ae2a127b-3a86-45d1-8e4a-10557a96a711" (UID: "ae2a127b-3a86-45d1-8e4a-10557a96a711"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:35.881364 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.881250    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-workload-socket" pod "ae2a127b-3a86-45d1-8e4a-10557a96a711" (UID: "ae2a127b-3a86-45d1-8e4a-10557a96a711"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:35.881364 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.881289    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2a127b-3a86-45d1-8e4a-10557a96a711-istiod-ca-cert" pod "ae2a127b-3a86-45d1-8e4a-10557a96a711" (UID: "ae2a127b-3a86-45d1-8e4a-10557a96a711"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:44:35.881776 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.881414    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-workload-certs" pod "ae2a127b-3a86-45d1-8e4a-10557a96a711" (UID: "ae2a127b-3a86-45d1-8e4a-10557a96a711"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:35.881776 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.881548    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-data" pod "ae2a127b-3a86-45d1-8e4a-10557a96a711" (UID: "ae2a127b-3a86-45d1-8e4a-10557a96a711"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:35.881901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.881856    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2a127b-3a86-45d1-8e4a-10557a96a711-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:35.881901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.881872    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:35.881901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.881884    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:35.881901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.881893    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:35.881901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.881903    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:35.883154 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.883128    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2a127b-3a86-45d1-8e4a-10557a96a711-kube-api-access-w5276" pod "ae2a127b-3a86-45d1-8e4a-10557a96a711" (UID: "ae2a127b-3a86-45d1-8e4a-10557a96a711"). InnerVolumeSpecName "kube-api-access-w5276". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:35.883260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.883211    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-token" pod "ae2a127b-3a86-45d1-8e4a-10557a96a711" (UID: "ae2a127b-3a86-45d1-8e4a-10557a96a711"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:35.883955 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.883925    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-podinfo" pod "ae2a127b-3a86-45d1-8e4a-10557a96a711" (UID: "ae2a127b-3a86-45d1-8e4a-10557a96a711"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:44:35.884051 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.883931    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-envoy" pod "ae2a127b-3a86-45d1-8e4a-10557a96a711" (UID: "ae2a127b-3a86-45d1-8e4a-10557a96a711"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:35.982920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.982833    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:35.982920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.982866    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:35.982920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.982882    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5276\" (UniqueName: \"kubernetes.io/projected/ae2a127b-3a86-45d1-8e4a-10557a96a711-kube-api-access-w5276\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:35.982920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:35.982896    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2a127b-3a86-45d1-8e4a-10557a96a711-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:36.042183 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:36.042155    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"]
Oct 30 23:44:36.043364 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:44:36.043334    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412494b5_417a_488e_9ce4_9771d6458325.slice/crio-25324bb0854f554e48ba1f0d4ef0cd5361875c97bc621a6e307d692c7b732d6b WatchSource:0}: Error finding container 25324bb0854f554e48ba1f0d4ef0cd5361875c97bc621a6e307d692c7b732d6b: Status 404 returned error can't find the container with id 25324bb0854f554e48ba1f0d4ef0cd5361875c97bc621a6e307d692c7b732d6b
Oct 30 23:44:36.045967 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:36.045937    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:44:36.046024 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:36.046013    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:44:36.046067 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:36.046041    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:44:36.527384 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:36.527355    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"
Oct 30 23:44:36.527874 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:36.527382    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6" event={"ID":"ae2a127b-3a86-45d1-8e4a-10557a96a711","Type":"ContainerDied","Data":"38e3ebcb36a2b7d4d1e4105aa86f456cf1e3fe550f8cd4b19dac44371e579340"}
Oct 30 23:44:36.527874 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:36.527446    5078 scope.go:122] "RemoveContainer" containerID="21d276b2ca00ef3304604872f3206f84b51c0b3d12e0be882544e5757d26bcc9"
Oct 30 23:44:36.529336 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:36.529308    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c" event={"ID":"412494b5-417a-488e-9ce4-9771d6458325","Type":"ContainerStarted","Data":"347067e02bdf480b23781cb99d09e82cd306cecd5979478f5d61d6abc399addb"}
Oct 30 23:44:36.529471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:36.529353    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c" event={"ID":"412494b5-417a-488e-9ce4-9771d6458325","Type":"ContainerStarted","Data":"25324bb0854f554e48ba1f0d4ef0cd5361875c97bc621a6e307d692c7b732d6b"}
Oct 30 23:44:36.531433 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:36.531412    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:36.700392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:36.700287    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c" podStartSLOduration=1.700270765 podStartE2EDuration="1.700270765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:44:36.698399137 +0000 UTC m=+2262.586305490" watchObservedRunningTime="2026-10-30 23:44:36.700270765 +0000 UTC m=+2262.588177117"
Oct 30 23:44:36.829626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:36.828986    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"]
Oct 30 23:44:36.879688 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:36.879653    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0a7530-openshift-default-8c964f7df-26xz6"]
Oct 30 23:44:37.537931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:37.537891    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:37.539448 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:37.539423    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:37.540127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:37.540109    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:44:38.298722 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:38.298692    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae2a127b-3a86-45d1-8e4a-10557a96a711" path="/var/lib/kubelet/pods/ae2a127b-3a86-45d1-8e4a-10557a96a711/volumes"
Oct 30 23:44:38.754309 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:38.754222    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82"]
Oct 30 23:44:38.859469 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:38.859437    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm"]
Oct 30 23:44:42.276077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.272432    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k"]
Oct 30 23:44:42.276077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.273332    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ae2a127b-3a86-45d1-8e4a-10557a96a711" containerName="istio-proxy"
Oct 30 23:44:42.276077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.273355    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2a127b-3a86-45d1-8e4a-10557a96a711" containerName="istio-proxy"
Oct 30 23:44:42.276077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.273573    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2a127b-3a86-45d1-8e4a-10557a96a711" containerName="istio-proxy"
Oct 30 23:44:42.276077 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.273630    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2a127b-3a86-45d1-8e4a-10557a96a711" containerName="istio-proxy"
Oct 30 23:44:42.288481 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.288452    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k"
Oct 30 23:44:42.302085 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.302059    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k"]
Oct 30 23:44:42.340783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.340750    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:44:42.340783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.340749    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:44:42.344175 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.344150    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:44:42.344300 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.344198    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:44:42.426241 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.426212    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjlr8\" (UniqueName: \"kubernetes.io/projected/984223ba-f0b0-483c-9587-e2d6420dcfa0-kube-api-access-xjlr8\") pod \"maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k\" (UID: \"984223ba-f0b0-483c-9587-e2d6420dcfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k"
Oct 30 23:44:42.426457 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.426438    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984223ba-f0b0-483c-9587-e2d6420dcfa0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k\" (UID: \"984223ba-f0b0-483c-9587-e2d6420dcfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k"
Oct 30 23:44:42.426521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.426506    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984223ba-f0b0-483c-9587-e2d6420dcfa0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k\" (UID: \"984223ba-f0b0-483c-9587-e2d6420dcfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k"
Oct 30 23:44:42.527743 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.527672    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984223ba-f0b0-483c-9587-e2d6420dcfa0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k\" (UID: \"984223ba-f0b0-483c-9587-e2d6420dcfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k"
Oct 30 23:44:42.527743 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.527735    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984223ba-f0b0-483c-9587-e2d6420dcfa0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k\" (UID: \"984223ba-f0b0-483c-9587-e2d6420dcfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k"
Oct 30 23:44:42.527923 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.527792    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjlr8\" (UniqueName: \"kubernetes.io/projected/984223ba-f0b0-483c-9587-e2d6420dcfa0-kube-api-access-xjlr8\") pod \"maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k\" (UID: \"984223ba-f0b0-483c-9587-e2d6420dcfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k"
Oct 30 23:44:42.528442 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.528380    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984223ba-f0b0-483c-9587-e2d6420dcfa0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k\" (UID: \"984223ba-f0b0-483c-9587-e2d6420dcfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k"
Oct 30 23:44:42.530160 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.530137    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984223ba-f0b0-483c-9587-e2d6420dcfa0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k\" (UID: \"984223ba-f0b0-483c-9587-e2d6420dcfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k"
Oct 30 23:44:42.585589 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.585558    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjlr8\" (UniqueName: \"kubernetes.io/projected/984223ba-f0b0-483c-9587-e2d6420dcfa0-kube-api-access-xjlr8\") pod \"maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k\" (UID: \"984223ba-f0b0-483c-9587-e2d6420dcfa0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k"
Oct 30 23:44:42.604104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:42.604075    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k"
Oct 30 23:44:43.005938 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:43.005910    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k"]
Oct 30 23:44:43.006788 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:44:43.006761    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod984223ba_f0b0_483c_9587_e2d6420dcfa0.slice/crio-d483f52a1aec146d805846fbce95cd475e58e8afad9fe80e4496bca44622e9a0 WatchSource:0}: Error finding container d483f52a1aec146d805846fbce95cd475e58e8afad9fe80e4496bca44622e9a0: Status 404 returned error can't find the container with id d483f52a1aec146d805846fbce95cd475e58e8afad9fe80e4496bca44622e9a0
Oct 30 23:44:43.618209 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:43.618109    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k" event={"ID":"984223ba-f0b0-483c-9587-e2d6420dcfa0","Type":"ContainerStarted","Data":"adb018a3323611ec6c9ac7d78d378b2698547915bd131a24e00e628605e9e23d"}
Oct 30 23:44:43.618567 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:43.618208    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k" event={"ID":"984223ba-f0b0-483c-9587-e2d6420dcfa0","Type":"ContainerStarted","Data":"d483f52a1aec146d805846fbce95cd475e58e8afad9fe80e4496bca44622e9a0"}
Oct 30 23:44:43.733232 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:43.733178    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k" podStartSLOduration=1.733164687 podStartE2EDuration="1.733164687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:44:43.729742643 +0000 UTC m=+2269.617648996" watchObservedRunningTime="2026-10-30 23:44:43.733164687 +0000 UTC m=+2269.621071018"
Oct 30 23:44:43.755093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:43.755061    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82" podUID="6fca3a8d-ef11-4d95-ba93-bcafa869e2c1" containerName="payload-processing" containerID="cri-o://155b6086942f9dd6951d898325178e8b58cffc1ad7380d5c2779304a03dd83ff" gracePeriod=25
Oct 30 23:44:43.860028 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:43.859978    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm" podUID="b6b6065b-de7d-41b2-9733-42f56d923d59" containerName="payload-pre-processing" containerID="cri-o://636769bcd33bd31630837964dd6b66d948889f64bb604f3f309b3a3d5d3e66e3" gracePeriod=25
Oct 30 23:44:44.626781 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:44.626749    5078 generic.go:356] "Generic (PLEG): container finished" podID="984223ba-f0b0-483c-9587-e2d6420dcfa0" containerID="adb018a3323611ec6c9ac7d78d378b2698547915bd131a24e00e628605e9e23d" exitCode=0
Oct 30 23:44:44.627183 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:44.626830    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k" event={"ID":"984223ba-f0b0-483c-9587-e2d6420dcfa0","Type":"ContainerDied","Data":"adb018a3323611ec6c9ac7d78d378b2698547915bd131a24e00e628605e9e23d"}
Oct 30 23:44:45.538879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:45.505520    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"]
Oct 30 23:44:45.867535 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:45.867491    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm" podUID="b6b6065b-de7d-41b2-9733-42f56d923d59" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:44:45.872267 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:45.872238    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82" podUID="6fca3a8d-ef11-4d95-ba93-bcafa869e2c1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:44:45.995953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:45.995925    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k"
Oct 30 23:44:46.137238 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:46.137150    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/984223ba-f0b0-483c-9587-e2d6420dcfa0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984223ba-f0b0-483c-9587-e2d6420dcfa0-maas-api-xks-ca\") pod \"984223ba-f0b0-483c-9587-e2d6420dcfa0\" (UID: \"984223ba-f0b0-483c-9587-e2d6420dcfa0\") "
Oct 30 23:44:46.137415 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:46.137278    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/984223ba-f0b0-483c-9587-e2d6420dcfa0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984223ba-f0b0-483c-9587-e2d6420dcfa0-maas-api-openshift-ca\") pod \"984223ba-f0b0-483c-9587-e2d6420dcfa0\" (UID: \"984223ba-f0b0-483c-9587-e2d6420dcfa0\") "
Oct 30 23:44:46.137490 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:46.137441    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/984223ba-f0b0-483c-9587-e2d6420dcfa0-kube-api-access-xjlr8\" (UniqueName: \"kubernetes.io/projected/984223ba-f0b0-483c-9587-e2d6420dcfa0-kube-api-access-xjlr8\") pod \"984223ba-f0b0-483c-9587-e2d6420dcfa0\" (UID: \"984223ba-f0b0-483c-9587-e2d6420dcfa0\") "
Oct 30 23:44:46.138082 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:46.138052    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/984223ba-f0b0-483c-9587-e2d6420dcfa0-maas-api-openshift-ca" pod "984223ba-f0b0-483c-9587-e2d6420dcfa0" (UID: "984223ba-f0b0-483c-9587-e2d6420dcfa0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:44:46.140087 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:46.140055    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/984223ba-f0b0-483c-9587-e2d6420dcfa0-maas-api-xks-ca" pod "984223ba-f0b0-483c-9587-e2d6420dcfa0" (UID: "984223ba-f0b0-483c-9587-e2d6420dcfa0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:44:46.140401 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:46.140373    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/984223ba-f0b0-483c-9587-e2d6420dcfa0-kube-api-access-xjlr8" pod "984223ba-f0b0-483c-9587-e2d6420dcfa0" (UID: "984223ba-f0b0-483c-9587-e2d6420dcfa0"). InnerVolumeSpecName "kube-api-access-xjlr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:46.238871 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:46.238836    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjlr8\" (UniqueName: \"kubernetes.io/projected/984223ba-f0b0-483c-9587-e2d6420dcfa0-kube-api-access-xjlr8\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:46.238871 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:46.238872    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/984223ba-f0b0-483c-9587-e2d6420dcfa0-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:46.238871 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:46.238885    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/984223ba-f0b0-483c-9587-e2d6420dcfa0-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:46.641769 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:46.641740    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k"
Oct 30 23:44:46.641951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:46.641782    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k" event={"ID":"984223ba-f0b0-483c-9587-e2d6420dcfa0","Type":"ContainerDied","Data":"d483f52a1aec146d805846fbce95cd475e58e8afad9fe80e4496bca44622e9a0"}
Oct 30 23:44:46.641951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:46.641827    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d483f52a1aec146d805846fbce95cd475e58e8afad9fe80e4496bca44622e9a0"
Oct 30 23:44:46.696035 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:46.696002    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main" containerID="cri-o://c38856da170847c361c4e54e6af13d7806d17b2551b934eb28e37bbf650f9a6e" gracePeriod=45
Oct 30 23:44:46.698301 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:46.698273    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main" containerID="cri-o://be8a449bd71aa6d8cf193e4af762092bc44f1ab6d0b2aa6ab198d6bfba7d846d" gracePeriod=45
Oct 30 23:44:46.703243 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:46.703211    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main" containerID="cri-o://21116669f17b83f1ead244298927743d636f6ce239d0e880279bfb6ea9f31c5f" gracePeriod=45
Oct 30 23:44:47.088978 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.088958    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:44:47.133928 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.133902    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:44:47.153507 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.153480    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:44:47.170099 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.170021    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-tmp-dir\") pod \"07fffabb-6a6d-4ec5-9534-2c3af669a252\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") "
Oct 30 23:44:47.170247 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.170179    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07fffabb-6a6d-4ec5-9534-2c3af669a252-tls-certs\" (UniqueName: \"kubernetes.io/secret/07fffabb-6a6d-4ec5-9534-2c3af669a252-tls-certs\") pod \"07fffabb-6a6d-4ec5-9534-2c3af669a252\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") "
Oct 30 23:44:47.170247 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.170239    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-dshm\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-dshm\") pod \"07fffabb-6a6d-4ec5-9534-2c3af669a252\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") "
Oct 30 23:44:47.170376 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.170317    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07fffabb-6a6d-4ec5-9534-2c3af669a252-kube-api-access-98ggf\" (UniqueName: \"kubernetes.io/projected/07fffabb-6a6d-4ec5-9534-2c3af669a252-kube-api-access-98ggf\") pod \"07fffabb-6a6d-4ec5-9534-2c3af669a252\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") "
Oct 30 23:44:47.170376 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.170314    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-tmp-dir" pod "07fffabb-6a6d-4ec5-9534-2c3af669a252" (UID: "07fffabb-6a6d-4ec5-9534-2c3af669a252"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.170501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.170408    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-model-cache\") pod \"07fffabb-6a6d-4ec5-9534-2c3af669a252\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") "
Oct 30 23:44:47.170501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.170483    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-home\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-home\") pod \"07fffabb-6a6d-4ec5-9534-2c3af669a252\" (UID: \"07fffabb-6a6d-4ec5-9534-2c3af669a252\") "
Oct 30 23:44:47.170773 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.170733    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-model-cache" pod "07fffabb-6a6d-4ec5-9534-2c3af669a252" (UID: "07fffabb-6a6d-4ec5-9534-2c3af669a252"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.170982 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.170958    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-home" pod "07fffabb-6a6d-4ec5-9534-2c3af669a252" (UID: "07fffabb-6a6d-4ec5-9534-2c3af669a252"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.171302 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.171279    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.171391 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.171309    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.171391 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.171350    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.173066 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.173045    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fffabb-6a6d-4ec5-9534-2c3af669a252-kube-api-access-98ggf" pod "07fffabb-6a6d-4ec5-9534-2c3af669a252" (UID: "07fffabb-6a6d-4ec5-9534-2c3af669a252"). InnerVolumeSpecName "kube-api-access-98ggf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:47.177951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.177897    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-dshm" pod "07fffabb-6a6d-4ec5-9534-2c3af669a252" (UID: "07fffabb-6a6d-4ec5-9534-2c3af669a252"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.179815 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.179790    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07fffabb-6a6d-4ec5-9534-2c3af669a252-tls-certs" pod "07fffabb-6a6d-4ec5-9534-2c3af669a252" (UID: "07fffabb-6a6d-4ec5-9534-2c3af669a252"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:44:47.272023 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.271993    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-model-cache\") pod \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") "
Oct 30 23:44:47.272191 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.272060    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-tmp-dir\") pod \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") "
Oct 30 23:44:47.272191 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.272092    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-dshm\") pod \"10cebe5f-5bd1-4d37-850f-a76663e0571e\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") "
Oct 30 23:44:47.272191 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.272159    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10cebe5f-5bd1-4d37-850f-a76663e0571e-kube-api-access-tg7vj\" (UniqueName: \"kubernetes.io/projected/10cebe5f-5bd1-4d37-850f-a76663e0571e-kube-api-access-tg7vj\") pod \"10cebe5f-5bd1-4d37-850f-a76663e0571e\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") "
Oct 30 23:44:47.272191 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.272184    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-model-cache\") pod \"10cebe5f-5bd1-4d37-850f-a76663e0571e\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") "
Oct 30 23:44:47.272475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.272213    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-tmp-dir\") pod \"10cebe5f-5bd1-4d37-850f-a76663e0571e\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") "
Oct 30 23:44:47.272475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.272232    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-model-cache" pod "ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" (UID: "ac32e8b3-eb6f-4ee7-9f59-a548422bc43a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.272475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.272244    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-home\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-home\") pod \"10cebe5f-5bd1-4d37-850f-a76663e0571e\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") "
Oct 30 23:44:47.272475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.272339    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-tmp-dir" pod "ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" (UID: "ac32e8b3-eb6f-4ee7-9f59-a548422bc43a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.272475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.272373    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-home\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-home\") pod \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") "
Oct 30 23:44:47.272475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.272435    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-tls-certs\") pod \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") "
Oct 30 23:44:47.272861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.272484    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-home" pod "10cebe5f-5bd1-4d37-850f-a76663e0571e" (UID: "10cebe5f-5bd1-4d37-850f-a76663e0571e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.272861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.272501    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-model-cache" pod "10cebe5f-5bd1-4d37-850f-a76663e0571e" (UID: "10cebe5f-5bd1-4d37-850f-a76663e0571e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.272861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.272520    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10cebe5f-5bd1-4d37-850f-a76663e0571e-tls-certs\" (UniqueName: \"kubernetes.io/secret/10cebe5f-5bd1-4d37-850f-a76663e0571e-tls-certs\") pod \"10cebe5f-5bd1-4d37-850f-a76663e0571e\" (UID: \"10cebe5f-5bd1-4d37-850f-a76663e0571e\") "
Oct 30 23:44:47.272861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.272512    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-tmp-dir" pod "10cebe5f-5bd1-4d37-850f-a76663e0571e" (UID: "10cebe5f-5bd1-4d37-850f-a76663e0571e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.272861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.272616    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-dshm\") pod \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") "
Oct 30 23:44:47.272861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.272645    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-home" pod "ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" (UID: "ac32e8b3-eb6f-4ee7-9f59-a548422bc43a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.272861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.272721    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-kube-api-access-j2vw7\" (UniqueName: \"kubernetes.io/projected/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-kube-api-access-j2vw7\") pod \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\" (UID: \"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a\") "
Oct 30 23:44:47.273705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.273512    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.273705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.273543    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.273705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.273558    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07fffabb-6a6d-4ec5-9534-2c3af669a252-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.273705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.273573    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.273705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.273588    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07fffabb-6a6d-4ec5-9534-2c3af669a252-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.273705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.273635    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.273705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.273651    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98ggf\" (UniqueName: \"kubernetes.io/projected/07fffabb-6a6d-4ec5-9534-2c3af669a252-kube-api-access-98ggf\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.273705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.273667    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.273705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.273681    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.274947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.274916    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-dshm" pod "10cebe5f-5bd1-4d37-850f-a76663e0571e" (UID: "10cebe5f-5bd1-4d37-850f-a76663e0571e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.275058 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.275044    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-tls-certs" pod "ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" (UID: "ac32e8b3-eb6f-4ee7-9f59-a548422bc43a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:44:47.275131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.275111    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10cebe5f-5bd1-4d37-850f-a76663e0571e-kube-api-access-tg7vj" pod "10cebe5f-5bd1-4d37-850f-a76663e0571e" (UID: "10cebe5f-5bd1-4d37-850f-a76663e0571e"). InnerVolumeSpecName "kube-api-access-tg7vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:47.275131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.275104    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-dshm" pod "ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" (UID: "ac32e8b3-eb6f-4ee7-9f59-a548422bc43a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:44:47.275266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.275245    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10cebe5f-5bd1-4d37-850f-a76663e0571e-tls-certs" pod "10cebe5f-5bd1-4d37-850f-a76663e0571e" (UID: "10cebe5f-5bd1-4d37-850f-a76663e0571e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:44:47.275497 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.275482    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-kube-api-access-j2vw7" pod "ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" (UID: "ac32e8b3-eb6f-4ee7-9f59-a548422bc43a"). InnerVolumeSpecName "kube-api-access-j2vw7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:44:47.374590 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.374556    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/10cebe5f-5bd1-4d37-850f-a76663e0571e-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.374777 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.374586    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg7vj\" (UniqueName: \"kubernetes.io/projected/10cebe5f-5bd1-4d37-850f-a76663e0571e-kube-api-access-tg7vj\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.374777 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.374633    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.374777 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.374647    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/10cebe5f-5bd1-4d37-850f-a76663e0571e-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.374777 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.374659    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.374777 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.374668    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2vw7\" (UniqueName: \"kubernetes.io/projected/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a-kube-api-access-j2vw7\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:44:47.592868 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:44:47.592832    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:44:47.650155 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.650123    5078 generic.go:356] "Generic (PLEG): container finished" podID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerID="21116669f17b83f1ead244298927743d636f6ce239d0e880279bfb6ea9f31c5f" exitCode=0
Oct 30 23:44:47.650352 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.650169    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"
Oct 30 23:44:47.650352 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.650210    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" event={"ID":"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a","Type":"ContainerDied","Data":"21116669f17b83f1ead244298927743d636f6ce239d0e880279bfb6ea9f31c5f"}
Oct 30 23:44:47.650352 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.650255    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc" event={"ID":"ac32e8b3-eb6f-4ee7-9f59-a548422bc43a","Type":"ContainerDied","Data":"1e8b000f2e71e55515d4b48c3bd5b2a63fa7d81d20ddc1d01c242148cba2ca87"}
Oct 30 23:44:47.650352 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.650280    5078 scope.go:122] "RemoveContainer" containerID="21116669f17b83f1ead244298927743d636f6ce239d0e880279bfb6ea9f31c5f"
Oct 30 23:44:47.652215 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.652171    5078 generic.go:356] "Generic (PLEG): container finished" podID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerID="c38856da170847c361c4e54e6af13d7806d17b2551b934eb28e37bbf650f9a6e" exitCode=0
Oct 30 23:44:47.652352 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.652240    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"
Oct 30 23:44:47.652352 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.652291    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" event={"ID":"07fffabb-6a6d-4ec5-9534-2c3af669a252","Type":"ContainerDied","Data":"c38856da170847c361c4e54e6af13d7806d17b2551b934eb28e37bbf650f9a6e"}
Oct 30 23:44:47.652352 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.652331    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz" event={"ID":"07fffabb-6a6d-4ec5-9534-2c3af669a252","Type":"ContainerDied","Data":"8ae1b2e3b92c75f9120adac53322ac0ec5741df72b5b1f2f5b8d6b70f1a640ad"}
Oct 30 23:44:47.654351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.654330    5078 generic.go:356] "Generic (PLEG): container finished" podID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerID="be8a449bd71aa6d8cf193e4af762092bc44f1ab6d0b2aa6ab198d6bfba7d846d" exitCode=0
Oct 30 23:44:47.654478 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.654382    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"
Oct 30 23:44:47.654478 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.654429    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" event={"ID":"10cebe5f-5bd1-4d37-850f-a76663e0571e","Type":"ContainerDied","Data":"be8a449bd71aa6d8cf193e4af762092bc44f1ab6d0b2aa6ab198d6bfba7d846d"}
Oct 30 23:44:47.654478 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.654458    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566" event={"ID":"10cebe5f-5bd1-4d37-850f-a76663e0571e","Type":"ContainerDied","Data":"ba2f02af332d4d8a4c95e8a2363944ddf7860b1b29211948287f317d0d340ddc"}
Oct 30 23:44:47.668647 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.668630    5078 scope.go:122] "RemoveContainer" containerID="21116669f17b83f1ead244298927743d636f6ce239d0e880279bfb6ea9f31c5f"
Oct 30 23:44:47.668912 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:44:47.668893    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21116669f17b83f1ead244298927743d636f6ce239d0e880279bfb6ea9f31c5f\": container with ID starting with 21116669f17b83f1ead244298927743d636f6ce239d0e880279bfb6ea9f31c5f not found: ID does not exist" containerID="21116669f17b83f1ead244298927743d636f6ce239d0e880279bfb6ea9f31c5f"
Oct 30 23:44:47.668973 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.668921    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21116669f17b83f1ead244298927743d636f6ce239d0e880279bfb6ea9f31c5f"} err="failed to get container status \"21116669f17b83f1ead244298927743d636f6ce239d0e880279bfb6ea9f31c5f\": rpc error: code = NotFound desc = could not find container \"21116669f17b83f1ead244298927743d636f6ce239d0e880279bfb6ea9f31c5f\": container with ID starting with 21116669f17b83f1ead244298927743d636f6ce239d0e880279bfb6ea9f31c5f not found: ID does not exist"
Oct 30 23:44:47.668973 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.668947    5078 scope.go:122] "RemoveContainer" containerID="c38856da170847c361c4e54e6af13d7806d17b2551b934eb28e37bbf650f9a6e"
Oct 30 23:44:47.677865 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.677850    5078 scope.go:122] "RemoveContainer" containerID="c38856da170847c361c4e54e6af13d7806d17b2551b934eb28e37bbf650f9a6e"
Oct 30 23:44:47.678141 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:44:47.678122    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c38856da170847c361c4e54e6af13d7806d17b2551b934eb28e37bbf650f9a6e\": container with ID starting with c38856da170847c361c4e54e6af13d7806d17b2551b934eb28e37bbf650f9a6e not found: ID does not exist" containerID="c38856da170847c361c4e54e6af13d7806d17b2551b934eb28e37bbf650f9a6e"
Oct 30 23:44:47.678199 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.678145    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c38856da170847c361c4e54e6af13d7806d17b2551b934eb28e37bbf650f9a6e"} err="failed to get container status \"c38856da170847c361c4e54e6af13d7806d17b2551b934eb28e37bbf650f9a6e\": rpc error: code = NotFound desc = could not find container \"c38856da170847c361c4e54e6af13d7806d17b2551b934eb28e37bbf650f9a6e\": container with ID starting with c38856da170847c361c4e54e6af13d7806d17b2551b934eb28e37bbf650f9a6e not found: ID does not exist"
Oct 30 23:44:47.678199 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.678160    5078 scope.go:122] "RemoveContainer" containerID="be8a449bd71aa6d8cf193e4af762092bc44f1ab6d0b2aa6ab198d6bfba7d846d"
Oct 30 23:44:47.686745 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.686725    5078 scope.go:122] "RemoveContainer" containerID="be8a449bd71aa6d8cf193e4af762092bc44f1ab6d0b2aa6ab198d6bfba7d846d"
Oct 30 23:44:47.687003 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:44:47.686982    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be8a449bd71aa6d8cf193e4af762092bc44f1ab6d0b2aa6ab198d6bfba7d846d\": container with ID starting with be8a449bd71aa6d8cf193e4af762092bc44f1ab6d0b2aa6ab198d6bfba7d846d not found: ID does not exist" containerID="be8a449bd71aa6d8cf193e4af762092bc44f1ab6d0b2aa6ab198d6bfba7d846d"
Oct 30 23:44:47.687141 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.687007    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8a449bd71aa6d8cf193e4af762092bc44f1ab6d0b2aa6ab198d6bfba7d846d"} err="failed to get container status \"be8a449bd71aa6d8cf193e4af762092bc44f1ab6d0b2aa6ab198d6bfba7d846d\": rpc error: code = NotFound desc = could not find container \"be8a449bd71aa6d8cf193e4af762092bc44f1ab6d0b2aa6ab198d6bfba7d846d\": container with ID starting with be8a449bd71aa6d8cf193e4af762092bc44f1ab6d0b2aa6ab198d6bfba7d846d not found: ID does not exist"
Oct 30 23:44:47.814066 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.814035    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"]
Oct 30 23:44:47.863379 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.863305    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-057303/distinct-w0-057303-kserve-76dbfb9f6f-4f566"]
Oct 30 23:44:47.928465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.928426    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"]
Oct 30 23:44:47.931768 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:47.931746    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-057303/premium-w0-057303-kserve-567764f6b-6gssc"]
Oct 30 23:44:48.040138 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:48.040107    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"]
Oct 30 23:44:48.042584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:48.042563    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-057303/distinct-2-w0-057303-kserve-578886f754-qrxhz"]
Oct 30 23:44:48.299757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:48.299731    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" path="/var/lib/kubelet/pods/07fffabb-6a6d-4ec5-9534-2c3af669a252/volumes"
Oct 30 23:44:48.300149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:48.300136    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" path="/var/lib/kubelet/pods/10cebe5f-5bd1-4d37-850f-a76663e0571e/volumes"
Oct 30 23:44:48.300511 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:48.300499    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" path="/var/lib/kubelet/pods/ac32e8b3-eb6f-4ee7-9f59-a548422bc43a/volumes"
Oct 30 23:44:55.863245 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:55.863204    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82" podUID="6fca3a8d-ef11-4d95-ba93-bcafa869e2c1" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:44:55.863656 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:55.863319    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm" podUID="b6b6065b-de7d-41b2-9733-42f56d923d59" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:44:56.834258 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:56.834225    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs"]
Oct 30 23:44:56.940895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:56.940846    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl"]
Oct 30 23:44:57.324673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.322403    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd"]
Oct 30 23:44:57.324673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.323179    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ae2a127b-3a86-45d1-8e4a-10557a96a711" containerName="istio-proxy"
Oct 30 23:44:57.324673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.323197    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2a127b-3a86-45d1-8e4a-10557a96a711" containerName="istio-proxy"
Oct 30 23:44:57.324673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.323213    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main"
Oct 30 23:44:57.324673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.323220    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main"
Oct 30 23:44:57.324673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.323248    5078 cpu_manager.go:414] "containerMap: removing container" podUID="984223ba-f0b0-483c-9587-e2d6420dcfa0" containerName="revoke-keys"
Oct 30 23:44:57.324673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.323256    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="984223ba-f0b0-483c-9587-e2d6420dcfa0" containerName="revoke-keys"
Oct 30 23:44:57.324673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.323279    5078 cpu_manager.go:414] "containerMap: removing container" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main"
Oct 30 23:44:57.324673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.323286    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main"
Oct 30 23:44:57.324673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.323320    5078 cpu_manager.go:414] "containerMap: removing container" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main"
Oct 30 23:44:57.324673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.323326    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main"
Oct 30 23:44:57.324673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.323434    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="984223ba-f0b0-483c-9587-e2d6420dcfa0" containerName="revoke-keys"
Oct 30 23:44:57.324673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.323447    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="07fffabb-6a6d-4ec5-9534-2c3af669a252" containerName="main"
Oct 30 23:44:57.324673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.323459    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="10cebe5f-5bd1-4d37-850f-a76663e0571e" containerName="main"
Oct 30 23:44:57.324673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.323469    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac32e8b3-eb6f-4ee7-9f59-a548422bc43a" containerName="main"
Oct 30 23:44:57.343252 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.343211    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd"
Oct 30 23:44:57.373260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.371879    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8"]
Oct 30 23:44:57.376410 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.376373    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8"
Oct 30 23:44:57.378127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.378103    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd"]
Oct 30 23:44:57.452629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.451984    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8"]
Oct 30 23:44:57.452629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.452204    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:44:57.453633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.453482    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:44:57.456171 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.453590    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:44:57.456625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.456344    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:44:57.460283 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.460255    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b9d2863-gwll8\" (UID: \"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8"
Oct 30 23:44:57.460393 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.460319    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwjth\" (UniqueName: \"kubernetes.io/projected/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-kube-api-access-jwjth\") pod \"maas-api-revoke-sub-6b9d2863-gwll8\" (UID: \"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8"
Oct 30 23:44:57.460393 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.460357    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b9d2863-gwll8\" (UID: \"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8"
Oct 30 23:44:57.561101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.561070    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxpp5\" (UniqueName: \"kubernetes.io/projected/b96c7b9a-2055-4654-b26c-d370869a2f1d-kube-api-access-hxpp5\") pod \"maas-api-revoke-sub-ad4ef5bb-zlwzd\" (UID: \"b96c7b9a-2055-4654-b26c-d370869a2f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd"
Oct 30 23:44:57.561271 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.561137    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b9d2863-gwll8\" (UID: \"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8"
Oct 30 23:44:57.561271 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.561223    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwjth\" (UniqueName: \"kubernetes.io/projected/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-kube-api-access-jwjth\") pod \"maas-api-revoke-sub-6b9d2863-gwll8\" (UID: \"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8"
Oct 30 23:44:57.561271 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.561264    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b9d2863-gwll8\" (UID: \"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8"
Oct 30 23:44:57.561433 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.561413    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96c7b9a-2055-4654-b26c-d370869a2f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad4ef5bb-zlwzd\" (UID: \"b96c7b9a-2055-4654-b26c-d370869a2f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd"
Oct 30 23:44:57.561717 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.561696    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96c7b9a-2055-4654-b26c-d370869a2f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad4ef5bb-zlwzd\" (UID: \"b96c7b9a-2055-4654-b26c-d370869a2f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd"
Oct 30 23:44:57.561888 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.561867    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b9d2863-gwll8\" (UID: \"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8"
Oct 30 23:44:57.563880 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.563861    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b9d2863-gwll8\" (UID: \"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8"
Oct 30 23:44:57.618025 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.617964    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwjth\" (UniqueName: \"kubernetes.io/projected/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-kube-api-access-jwjth\") pod \"maas-api-revoke-sub-6b9d2863-gwll8\" (UID: \"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8"
Oct 30 23:44:57.668616 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.663774    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96c7b9a-2055-4654-b26c-d370869a2f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad4ef5bb-zlwzd\" (UID: \"b96c7b9a-2055-4654-b26c-d370869a2f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd"
Oct 30 23:44:57.668616 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.663881    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96c7b9a-2055-4654-b26c-d370869a2f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad4ef5bb-zlwzd\" (UID: \"b96c7b9a-2055-4654-b26c-d370869a2f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd"
Oct 30 23:44:57.668616 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.663944    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxpp5\" (UniqueName: \"kubernetes.io/projected/b96c7b9a-2055-4654-b26c-d370869a2f1d-kube-api-access-hxpp5\") pod \"maas-api-revoke-sub-ad4ef5bb-zlwzd\" (UID: \"b96c7b9a-2055-4654-b26c-d370869a2f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd"
Oct 30 23:44:57.668616 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.665670    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96c7b9a-2055-4654-b26c-d370869a2f1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad4ef5bb-zlwzd\" (UID: \"b96c7b9a-2055-4654-b26c-d370869a2f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd"
Oct 30 23:44:57.676622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.672754    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96c7b9a-2055-4654-b26c-d370869a2f1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad4ef5bb-zlwzd\" (UID: \"b96c7b9a-2055-4654-b26c-d370869a2f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd"
Oct 30 23:44:57.722639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.722560    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxpp5\" (UniqueName: \"kubernetes.io/projected/b96c7b9a-2055-4654-b26c-d370869a2f1d-kube-api-access-hxpp5\") pod \"maas-api-revoke-sub-ad4ef5bb-zlwzd\" (UID: \"b96c7b9a-2055-4654-b26c-d370869a2f1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd"
Oct 30 23:44:57.760257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.760231    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8"
Oct 30 23:44:57.979883 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:57.979681    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd"
Oct 30 23:44:58.139935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:58.139904    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8"]
Oct 30 23:44:58.431619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:58.417428    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd"]
Oct 30 23:44:58.764652 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:58.764588    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8" event={"ID":"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4","Type":"ContainerStarted","Data":"cdfadb54f6a0e13ed6806256c77309d613624dd94d4587d67502c0111a9d4840"}
Oct 30 23:44:58.764849 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:58.764694    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8" event={"ID":"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4","Type":"ContainerStarted","Data":"1767b77086d0df5c7c80433716fc535e11ead391a635a3208b4bc2dfd6a414d3"}
Oct 30 23:44:58.765976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:58.765946    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd" event={"ID":"b96c7b9a-2055-4654-b26c-d370869a2f1d","Type":"ContainerStarted","Data":"a0aa1bc607c9769807e2031c3f6a449e383f9382e654645fd9920bceb42cb259"}
Oct 30 23:44:58.877738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:58.877679    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8" podStartSLOduration=1.877663139 podStartE2EDuration="1.877663139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:44:58.876972733 +0000 UTC m=+2284.764879110" watchObservedRunningTime="2026-10-30 23:44:58.877663139 +0000 UTC m=+2284.765569486"
Oct 30 23:44:59.702954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:59.702901    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg"]
Oct 30 23:44:59.705693 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:59.705670    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-55666687-5jplg"]
Oct 30 23:44:59.775191 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:59.775124    5078 generic.go:356] "Generic (PLEG): container finished" podID="b96c7b9a-2055-4654-b26c-d370869a2f1d" containerID="97367d30d0aab917ec6fc6073b9a509ab30f319ea5efe616d63d647ecbd364e0" exitCode=0
Oct 30 23:44:59.775319 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:59.775202    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd" event={"ID":"b96c7b9a-2055-4654-b26c-d370869a2f1d","Type":"ContainerDied","Data":"97367d30d0aab917ec6fc6073b9a509ab30f319ea5efe616d63d647ecbd364e0"}
Oct 30 23:44:59.776919 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:59.776895    5078 generic.go:356] "Generic (PLEG): container finished" podID="7cfecc2c-1a44-4f1d-828f-69f08b03a3f4" containerID="cdfadb54f6a0e13ed6806256c77309d613624dd94d4587d67502c0111a9d4840" exitCode=0
Oct 30 23:44:59.777049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:44:59.776920    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8" event={"ID":"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4","Type":"ContainerDied","Data":"cdfadb54f6a0e13ed6806256c77309d613624dd94d4587d67502c0111a9d4840"}
Oct 30 23:45:00.158734 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.158698    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr"]
Oct 30 23:45:00.163052 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.163026    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq"]
Oct 30 23:45:00.163187 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.163127    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr"
Oct 30 23:45:00.166517 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.166493    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8"]
Oct 30 23:45:00.166678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.166644    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq"
Oct 30 23:45:00.170816 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.170799    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn"]
Oct 30 23:45:00.170943 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.170925    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8"
Oct 30 23:45:00.174479 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.174460    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn"
Oct 30 23:45:00.191114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.191076    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5"]
Oct 30 23:45:00.195607 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.195576    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79"]
Oct 30 23:45:00.195707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.195695    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5"
Oct 30 23:45:00.200905 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.200887    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79"
Oct 30 23:45:00.215050 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.215032    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977fb1cb-c631-476a-8a27-373ce7cace0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5\" (UID: \"977fb1cb-c631-476a-8a27-373ce7cace0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5"
Oct 30 23:45:00.215152 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.215137    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0cd089-df4b-4d76-854b-217b6c93b42c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq\" (UID: \"ca0cd089-df4b-4d76-854b-217b6c93b42c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq"
Oct 30 23:45:00.215204 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.215176    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3349e30c-c66e-4f0f-b640-f0d1d0c50588-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8\" (UID: \"3349e30c-c66e-4f0f-b640-f0d1d0c50588\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8"
Oct 30 23:45:00.215274 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.215261    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acddcfa4-c6f5-413d-a172-74ec4ab2433f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890065-qfd79\" (UID: \"acddcfa4-c6f5-413d-a172-74ec4ab2433f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79"
Oct 30 23:45:00.215326 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.215284    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm8mn\" (UniqueName: \"kubernetes.io/projected/977fb1cb-c631-476a-8a27-373ce7cace0a-kube-api-access-mm8mn\") pod \"maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5\" (UID: \"977fb1cb-c631-476a-8a27-373ce7cace0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5"
Oct 30 23:45:00.215326 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.215311    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53dc275c-99f8-40a6-ae1e-7acb22c8718c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr\" (UID: \"53dc275c-99f8-40a6-ae1e-7acb22c8718c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr"
Oct 30 23:45:00.215426 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.215412    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9htrv\" (UniqueName: \"kubernetes.io/projected/00a24baa-07c5-4252-a4e9-8ca2dda0f829-kube-api-access-9htrv\") pod \"maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn\" (UID: \"00a24baa-07c5-4252-a4e9-8ca2dda0f829\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn"
Oct 30 23:45:00.215468 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.215455    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qrbd\" (UniqueName: \"kubernetes.io/projected/ca0cd089-df4b-4d76-854b-217b6c93b42c-kube-api-access-5qrbd\") pod \"maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq\" (UID: \"ca0cd089-df4b-4d76-854b-217b6c93b42c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq"
Oct 30 23:45:00.215514 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.215479    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wts2\" (UniqueName: \"kubernetes.io/projected/acddcfa4-c6f5-413d-a172-74ec4ab2433f-kube-api-access-4wts2\") pod \"maas-api-key-cleanup-29890065-qfd79\" (UID: \"acddcfa4-c6f5-413d-a172-74ec4ab2433f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79"
Oct 30 23:45:00.215564 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.215511    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnj97\" (UniqueName: \"kubernetes.io/projected/3349e30c-c66e-4f0f-b640-f0d1d0c50588-kube-api-access-mnj97\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8\" (UID: \"3349e30c-c66e-4f0f-b640-f0d1d0c50588\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8"
Oct 30 23:45:00.215627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.215584    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbblr\" (UniqueName: \"kubernetes.io/projected/53dc275c-99f8-40a6-ae1e-7acb22c8718c-kube-api-access-hbblr\") pod \"maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr\" (UID: \"53dc275c-99f8-40a6-ae1e-7acb22c8718c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr"
Oct 30 23:45:00.215669 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.215644    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a24baa-07c5-4252-a4e9-8ca2dda0f829-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn\" (UID: \"00a24baa-07c5-4252-a4e9-8ca2dda0f829\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn"
Oct 30 23:45:00.216194 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.216174    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp"]
Oct 30 23:45:00.220536 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.220517    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr"]
Oct 30 23:45:00.220652 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.220588    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp"
Oct 30 23:45:00.259325 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.259253    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq"]
Oct 30 23:45:00.264889 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.264854    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8"]
Oct 30 23:45:00.277156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.277136    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5"]
Oct 30 23:45:00.282991 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.280762    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79"]
Oct 30 23:45:00.283386 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.283363    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp"]
Oct 30 23:45:00.298912 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.298888    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bfd60b4-9b7c-41c2-ba66-a60640cf554e" path="/var/lib/kubelet/pods/7bfd60b4-9b7c-41c2-ba66-a60640cf554e/volumes"
Oct 30 23:45:00.317232 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.317211    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acddcfa4-c6f5-413d-a172-74ec4ab2433f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890065-qfd79\" (UID: \"acddcfa4-c6f5-413d-a172-74ec4ab2433f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79"
Oct 30 23:45:00.317370 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.317243    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm8mn\" (UniqueName: \"kubernetes.io/projected/977fb1cb-c631-476a-8a27-373ce7cace0a-kube-api-access-mm8mn\") pod \"maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5\" (UID: \"977fb1cb-c631-476a-8a27-373ce7cace0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5"
Oct 30 23:45:00.317370 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.317272    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53dc275c-99f8-40a6-ae1e-7acb22c8718c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr\" (UID: \"53dc275c-99f8-40a6-ae1e-7acb22c8718c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr"
Oct 30 23:45:00.317370 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.317321    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv8xm\" (UniqueName: \"kubernetes.io/projected/945e980d-0501-49a4-bad0-4af78c316441-kube-api-access-hv8xm\") pod \"maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp\" (UID: \"945e980d-0501-49a4-bad0-4af78c316441\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp"
Oct 30 23:45:00.317370 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.317358    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9htrv\" (UniqueName: \"kubernetes.io/projected/00a24baa-07c5-4252-a4e9-8ca2dda0f829-kube-api-access-9htrv\") pod \"maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn\" (UID: \"00a24baa-07c5-4252-a4e9-8ca2dda0f829\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn"
Oct 30 23:45:00.317629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.317392    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qrbd\" (UniqueName: \"kubernetes.io/projected/ca0cd089-df4b-4d76-854b-217b6c93b42c-kube-api-access-5qrbd\") pod \"maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq\" (UID: \"ca0cd089-df4b-4d76-854b-217b6c93b42c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq"
Oct 30 23:45:00.317629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.317424    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wts2\" (UniqueName: \"kubernetes.io/projected/acddcfa4-c6f5-413d-a172-74ec4ab2433f-kube-api-access-4wts2\") pod \"maas-api-key-cleanup-29890065-qfd79\" (UID: \"acddcfa4-c6f5-413d-a172-74ec4ab2433f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79"
Oct 30 23:45:00.317629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.317458    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/945e980d-0501-49a4-bad0-4af78c316441-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp\" (UID: \"945e980d-0501-49a4-bad0-4af78c316441\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp"
Oct 30 23:45:00.317629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.317500    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnj97\" (UniqueName: \"kubernetes.io/projected/3349e30c-c66e-4f0f-b640-f0d1d0c50588-kube-api-access-mnj97\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8\" (UID: \"3349e30c-c66e-4f0f-b640-f0d1d0c50588\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8"
Oct 30 23:45:00.317629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.317532    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbblr\" (UniqueName: \"kubernetes.io/projected/53dc275c-99f8-40a6-ae1e-7acb22c8718c-kube-api-access-hbblr\") pod \"maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr\" (UID: \"53dc275c-99f8-40a6-ae1e-7acb22c8718c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr"
Oct 30 23:45:00.317629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.317557    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a24baa-07c5-4252-a4e9-8ca2dda0f829-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn\" (UID: \"00a24baa-07c5-4252-a4e9-8ca2dda0f829\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn"
Oct 30 23:45:00.317629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.317614    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977fb1cb-c631-476a-8a27-373ce7cace0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5\" (UID: \"977fb1cb-c631-476a-8a27-373ce7cace0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5"
Oct 30 23:45:00.318012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.317732    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0cd089-df4b-4d76-854b-217b6c93b42c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq\" (UID: \"ca0cd089-df4b-4d76-854b-217b6c93b42c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq"
Oct 30 23:45:00.318012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.317812    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3349e30c-c66e-4f0f-b640-f0d1d0c50588-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8\" (UID: \"3349e30c-c66e-4f0f-b640-f0d1d0c50588\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8"
Oct 30 23:45:00.318177 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.318071    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acddcfa4-c6f5-413d-a172-74ec4ab2433f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890065-qfd79\" (UID: \"acddcfa4-c6f5-413d-a172-74ec4ab2433f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79"
Oct 30 23:45:00.318177 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.318085    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53dc275c-99f8-40a6-ae1e-7acb22c8718c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr\" (UID: \"53dc275c-99f8-40a6-ae1e-7acb22c8718c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr"
Oct 30 23:45:00.318707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.318681    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0cd089-df4b-4d76-854b-217b6c93b42c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq\" (UID: \"ca0cd089-df4b-4d76-854b-217b6c93b42c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq"
Oct 30 23:45:00.318792 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.318726    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3349e30c-c66e-4f0f-b640-f0d1d0c50588-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8\" (UID: \"3349e30c-c66e-4f0f-b640-f0d1d0c50588\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8"
Oct 30 23:45:00.318836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.318810    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977fb1cb-c631-476a-8a27-373ce7cace0a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5\" (UID: \"977fb1cb-c631-476a-8a27-373ce7cace0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5"
Oct 30 23:45:00.318876 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.318819    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a24baa-07c5-4252-a4e9-8ca2dda0f829-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn\" (UID: \"00a24baa-07c5-4252-a4e9-8ca2dda0f829\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn"
Oct 30 23:45:00.328005 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.327984    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn"]
Oct 30 23:45:00.374385 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.374354    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wts2\" (UniqueName: \"kubernetes.io/projected/acddcfa4-c6f5-413d-a172-74ec4ab2433f-kube-api-access-4wts2\") pod \"maas-api-key-cleanup-29890065-qfd79\" (UID: \"acddcfa4-c6f5-413d-a172-74ec4ab2433f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79"
Oct 30 23:45:00.374521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.374451    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbblr\" (UniqueName: \"kubernetes.io/projected/53dc275c-99f8-40a6-ae1e-7acb22c8718c-kube-api-access-hbblr\") pod \"maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr\" (UID: \"53dc275c-99f8-40a6-ae1e-7acb22c8718c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr"
Oct 30 23:45:00.374879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.374857    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm8mn\" (UniqueName: \"kubernetes.io/projected/977fb1cb-c631-476a-8a27-373ce7cace0a-kube-api-access-mm8mn\") pod \"maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5\" (UID: \"977fb1cb-c631-476a-8a27-373ce7cace0a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5"
Oct 30 23:45:00.375139 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.375118    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qrbd\" (UniqueName: \"kubernetes.io/projected/ca0cd089-df4b-4d76-854b-217b6c93b42c-kube-api-access-5qrbd\") pod \"maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq\" (UID: \"ca0cd089-df4b-4d76-854b-217b6c93b42c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq"
Oct 30 23:45:00.375502 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.375482    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnj97\" (UniqueName: \"kubernetes.io/projected/3349e30c-c66e-4f0f-b640-f0d1d0c50588-kube-api-access-mnj97\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8\" (UID: \"3349e30c-c66e-4f0f-b640-f0d1d0c50588\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8"
Oct 30 23:45:00.376102 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.376079    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9htrv\" (UniqueName: \"kubernetes.io/projected/00a24baa-07c5-4252-a4e9-8ca2dda0f829-kube-api-access-9htrv\") pod \"maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn\" (UID: \"00a24baa-07c5-4252-a4e9-8ca2dda0f829\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn"
Oct 30 23:45:00.419244 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.419175    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv8xm\" (UniqueName: \"kubernetes.io/projected/945e980d-0501-49a4-bad0-4af78c316441-kube-api-access-hv8xm\") pod \"maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp\" (UID: \"945e980d-0501-49a4-bad0-4af78c316441\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp"
Oct 30 23:45:00.419244 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.419238    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/945e980d-0501-49a4-bad0-4af78c316441-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp\" (UID: \"945e980d-0501-49a4-bad0-4af78c316441\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp"
Oct 30 23:45:00.419821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.419803    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/945e980d-0501-49a4-bad0-4af78c316441-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp\" (UID: \"945e980d-0501-49a4-bad0-4af78c316441\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp"
Oct 30 23:45:00.475684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.475659    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv8xm\" (UniqueName: \"kubernetes.io/projected/945e980d-0501-49a4-bad0-4af78c316441-kube-api-access-hv8xm\") pod \"maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp\" (UID: \"945e980d-0501-49a4-bad0-4af78c316441\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp"
Oct 30 23:45:00.483513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.483491    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr"
Oct 30 23:45:00.491942 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.491923    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq"
Oct 30 23:45:00.505496 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.505431    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8"
Oct 30 23:45:00.511404 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.511233    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn"
Oct 30 23:45:00.523447 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.523408    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5"
Oct 30 23:45:00.527319 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.527297    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79"
Oct 30 23:45:00.534917 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.534765    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp"
Oct 30 23:45:00.581029 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.579562    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main" containerID="cri-o://9c756a6e3dae78ac950916b5c5bf3277065b49907ea1da17f42b4eb9e0c4919f" gracePeriod=45
Oct 30 23:45:00.788426 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.788399    5078 generic.go:356] "Generic (PLEG): container finished" podID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerID="9c756a6e3dae78ac950916b5c5bf3277065b49907ea1da17f42b4eb9e0c4919f" exitCode=0
Oct 30 23:45:00.789028 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.788773    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" event={"ID":"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d","Type":"ContainerDied","Data":"9c756a6e3dae78ac950916b5c5bf3277065b49907ea1da17f42b4eb9e0c4919f"}
Oct 30 23:45:00.883134 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:45:00.883098    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53dc275c_99f8_40a6_ae1e_7acb22c8718c.slice/crio-88e63f92592247aaf1f1ea69b726904a5991f249da8351b153d04b4935a41dae WatchSource:0}: Error finding container 88e63f92592247aaf1f1ea69b726904a5991f249da8351b153d04b4935a41dae: Status 404 returned error can't find the container with id 88e63f92592247aaf1f1ea69b726904a5991f249da8351b153d04b4935a41dae
Oct 30 23:45:00.890034 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.889242    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr"]
Oct 30 23:45:00.923678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.923654    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:45:00.941270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.941251    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8"]
Oct 30 23:45:00.946562 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:00.946538    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq"]
Oct 30 23:45:01.012150 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.011854    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd"
Oct 30 23:45:01.016712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.016687    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8"
Oct 30 23:45:01.039369 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.038902    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-model-cache\") pod \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") "
Oct 30 23:45:01.039369 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.038959    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-tmp-dir\") pod \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") "
Oct 30 23:45:01.039369 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.038999    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b96c7b9a-2055-4654-b26c-d370869a2f1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96c7b9a-2055-4654-b26c-d370869a2f1d-maas-api-openshift-ca\") pod \"b96c7b9a-2055-4654-b26c-d370869a2f1d\" (UID: \"b96c7b9a-2055-4654-b26c-d370869a2f1d\") "
Oct 30 23:45:01.039369 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.039119    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-maas-api-openshift-ca\") pod \"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4\" (UID: \"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4\") "
Oct 30 23:45:01.039369 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.039152    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-tls-certs\") pod \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") "
Oct 30 23:45:01.039369 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.039191    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-home\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-home\") pod \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") "
Oct 30 23:45:01.039369 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.039235    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-dshm\") pod \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") "
Oct 30 23:45:01.039369 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.039267    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b96c7b9a-2055-4654-b26c-d370869a2f1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96c7b9a-2055-4654-b26c-d370869a2f1d-maas-api-xks-ca\") pod \"b96c7b9a-2055-4654-b26c-d370869a2f1d\" (UID: \"b96c7b9a-2055-4654-b26c-d370869a2f1d\") "
Oct 30 23:45:01.040010 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.039514    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-home" pod "cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" (UID: "cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:45:01.040010 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.039630    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-maas-api-openshift-ca" pod "7cfecc2c-1a44-4f1d-828f-69f08b03a3f4" (UID: "7cfecc2c-1a44-4f1d-828f-69f08b03a3f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:01.040132 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.040003    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-tmp-dir" pod "cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" (UID: "cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:45:01.040132 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.040086    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-model-cache" pod "cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" (UID: "cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:45:01.040234 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.040186    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96c7b9a-2055-4654-b26c-d370869a2f1d-maas-api-openshift-ca" pod "b96c7b9a-2055-4654-b26c-d370869a2f1d" (UID: "b96c7b9a-2055-4654-b26c-d370869a2f1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:01.042513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.040300    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-kube-api-access-jwjth\" (UniqueName: \"kubernetes.io/projected/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-kube-api-access-jwjth\") pod \"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4\" (UID: \"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4\") "
Oct 30 23:45:01.042513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.040367    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-kube-api-access-kdgr9\" (UniqueName: \"kubernetes.io/projected/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-kube-api-access-kdgr9\") pod \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\" (UID: \"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d\") "
Oct 30 23:45:01.042513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.040389    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b96c7b9a-2055-4654-b26c-d370869a2f1d-kube-api-access-hxpp5\" (UniqueName: \"kubernetes.io/projected/b96c7b9a-2055-4654-b26c-d370869a2f1d-kube-api-access-hxpp5\") pod \"b96c7b9a-2055-4654-b26c-d370869a2f1d\" (UID: \"b96c7b9a-2055-4654-b26c-d370869a2f1d\") "
Oct 30 23:45:01.042513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.040531    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-maas-api-xks-ca\") pod \"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4\" (UID: \"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4\") "
Oct 30 23:45:01.042513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.041315    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:01.042513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.041339    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:01.042881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.042520    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-tls-certs" pod "cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" (UID: "cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:45:01.042881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.041355    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:01.042881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.042653    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:01.042881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.042666    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b96c7b9a-2055-4654-b26c-d370869a2f1d-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:01.043010 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.042886    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-kube-api-access-kdgr9" pod "cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" (UID: "cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d"). InnerVolumeSpecName "kube-api-access-kdgr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:01.043383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.043355    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-kube-api-access-jwjth" pod "7cfecc2c-1a44-4f1d-828f-69f08b03a3f4" (UID: "7cfecc2c-1a44-4f1d-828f-69f08b03a3f4"). InnerVolumeSpecName "kube-api-access-jwjth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:01.043623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.043572    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-dshm" pod "cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" (UID: "cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:45:01.043954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.043928    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b96c7b9a-2055-4654-b26c-d370869a2f1d-maas-api-xks-ca" pod "b96c7b9a-2055-4654-b26c-d370869a2f1d" (UID: "b96c7b9a-2055-4654-b26c-d370869a2f1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:45:01.044830 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.044804    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96c7b9a-2055-4654-b26c-d370869a2f1d-kube-api-access-hxpp5" pod "b96c7b9a-2055-4654-b26c-d370869a2f1d" (UID: "b96c7b9a-2055-4654-b26c-d370869a2f1d"). InnerVolumeSpecName "kube-api-access-hxpp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:01.044968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.044949    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-maas-api-xks-ca" pod "7cfecc2c-1a44-4f1d-828f-69f08b03a3f4" (UID: "7cfecc2c-1a44-4f1d-828f-69f08b03a3f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:45:01.080979 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:45:01.080938    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a24baa_07c5_4252_a4e9_8ca2dda0f829.slice/crio-c37b04094aa4e9ed0e90d014ef2dc25487429ad02d7062084bb0c1c1b5daf1d1 WatchSource:0}: Error finding container c37b04094aa4e9ed0e90d014ef2dc25487429ad02d7062084bb0c1c1b5daf1d1: Status 404 returned error can't find the container with id c37b04094aa4e9ed0e90d014ef2dc25487429ad02d7062084bb0c1c1b5daf1d1
Oct 30 23:45:01.082073 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.082042    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn"]
Oct 30 23:45:01.144225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.144195    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:01.144225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.144222    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:01.144452 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.144234    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b96c7b9a-2055-4654-b26c-d370869a2f1d-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:01.144452 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.144246    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwjth\" (UniqueName: \"kubernetes.io/projected/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-kube-api-access-jwjth\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:01.144452 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.144256    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdgr9\" (UniqueName: \"kubernetes.io/projected/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d-kube-api-access-kdgr9\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:01.144452 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.144267    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxpp5\" (UniqueName: \"kubernetes.io/projected/b96c7b9a-2055-4654-b26c-d370869a2f1d-kube-api-access-hxpp5\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:01.144452 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.144277    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:01.167930 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.165648    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5"]
Oct 30 23:45:01.174014 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.173978    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79"]
Oct 30 23:45:01.239010 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.238986    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp"]
Oct 30 23:45:01.797521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.797473    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8" event={"ID":"3349e30c-c66e-4f0f-b640-f0d1d0c50588","Type":"ContainerStarted","Data":"a4c8767653311b14e46927b0e296ebaec0d91b2fc0eb4348e15d14892a0c7d50"}
Oct 30 23:45:01.798028 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.797571    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8" event={"ID":"3349e30c-c66e-4f0f-b640-f0d1d0c50588","Type":"ContainerStarted","Data":"c686e6ba6750517db9b457c378010444b9913cbb57852155324c5ce43067f4a9"}
Oct 30 23:45:01.799226 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.799199    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn" event={"ID":"00a24baa-07c5-4252-a4e9-8ca2dda0f829","Type":"ContainerStarted","Data":"f049223af5cd405b205dcfcae074efc659f1ae24507e9121302a64641af0f0e3"}
Oct 30 23:45:01.799338 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.799246    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn" event={"ID":"00a24baa-07c5-4252-a4e9-8ca2dda0f829","Type":"ContainerStarted","Data":"c37b04094aa4e9ed0e90d014ef2dc25487429ad02d7062084bb0c1c1b5daf1d1"}
Oct 30 23:45:01.800554 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.800537    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd"
Oct 30 23:45:01.800660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.800553    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd" event={"ID":"b96c7b9a-2055-4654-b26c-d370869a2f1d","Type":"ContainerDied","Data":"a0aa1bc607c9769807e2031c3f6a449e383f9382e654645fd9920bceb42cb259"}
Oct 30 23:45:01.800660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.800607    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0aa1bc607c9769807e2031c3f6a449e383f9382e654645fd9920bceb42cb259"
Oct 30 23:45:01.802270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.802246    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr" event={"ID":"53dc275c-99f8-40a6-ae1e-7acb22c8718c","Type":"ContainerStarted","Data":"5229467a22226be02c389cee0c0c1b60a5b10b32df486f6c0029c7a652f65ca5"}
Oct 30 23:45:01.802374 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.802280    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr" event={"ID":"53dc275c-99f8-40a6-ae1e-7acb22c8718c","Type":"ContainerStarted","Data":"88e63f92592247aaf1f1ea69b726904a5991f249da8351b153d04b4935a41dae"}
Oct 30 23:45:01.803647 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.803625    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5" event={"ID":"977fb1cb-c631-476a-8a27-373ce7cace0a","Type":"ContainerStarted","Data":"24c25302a45c93194fe3f921906144fc1d16ce3133468cbbc834f8c68f975383"}
Oct 30 23:45:01.805285 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.805259    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq" event={"ID":"ca0cd089-df4b-4d76-854b-217b6c93b42c","Type":"ContainerStarted","Data":"81cf70e9f48c35db0c085f1b48d5b28653d4da4f8c94392e71e98fedee38e71d"}
Oct 30 23:45:01.805387 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.805307    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq" event={"ID":"ca0cd089-df4b-4d76-854b-217b6c93b42c","Type":"ContainerStarted","Data":"d1f134832f0b2d3ee3e212d9dfc33246d25cd8852b3eaf3f48512ef707b60d16"}
Oct 30 23:45:01.806897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.806875    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79" event={"ID":"acddcfa4-c6f5-413d-a172-74ec4ab2433f","Type":"ContainerStarted","Data":"a09786b773d0eb9f83a4b006cd73937ccdadecdbcfa3616cdd02d2f1c8a88eb0"}
Oct 30 23:45:01.806999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.806909    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79" event={"ID":"acddcfa4-c6f5-413d-a172-74ec4ab2433f","Type":"ContainerStarted","Data":"fc55b089975cc379e03521fb70dbe490d9fcf6eb6c89e9ac98d6b5e2c11a39c6"}
Oct 30 23:45:01.808481 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.808460    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp" event={"ID":"945e980d-0501-49a4-bad0-4af78c316441","Type":"ContainerStarted","Data":"39edd45749487533bf70f99669f3689fa4df7c3369c61b79ad9b0bbb369090f3"}
Oct 30 23:45:01.808585 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.808504    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp" event={"ID":"945e980d-0501-49a4-bad0-4af78c316441","Type":"ContainerStarted","Data":"727cfefd598a55982d0a06a798373ee45be90ae3b4652743344047549582caca"}
Oct 30 23:45:01.809957 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.809928    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8"
Oct 30 23:45:01.810062 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.809923    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8" event={"ID":"7cfecc2c-1a44-4f1d-828f-69f08b03a3f4","Type":"ContainerDied","Data":"1767b77086d0df5c7c80433716fc535e11ead391a635a3208b4bc2dfd6a414d3"}
Oct 30 23:45:01.810062 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.810045    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1767b77086d0df5c7c80433716fc535e11ead391a635a3208b4bc2dfd6a414d3"
Oct 30 23:45:01.811255 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.811237    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r" event={"ID":"cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d","Type":"ContainerDied","Data":"ca1b990be04299c6b8d7916bd36d3f0e646e4aeb344a97572964c8608232eb38"}
Oct 30 23:45:01.811255 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.811247    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"
Oct 30 23:45:01.811394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.811271    5078 scope.go:122] "RemoveContainer" containerID="9c756a6e3dae78ac950916b5c5bf3277065b49907ea1da17f42b4eb9e0c4919f"
Oct 30 23:45:01.834524 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.834500    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs" podUID="b6416daa-de15-47d8-aca5-da1b417b6750" containerName="payload-processing" containerID="cri-o://1fefbc4127ab01f1682ca43b2837afd38f510d94b5135c29b32e843072e0851d" gracePeriod=25
Oct 30 23:45:01.909748 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.909656    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8" podStartSLOduration=1.9096383540000001 podStartE2EDuration="1.909638354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:01.90931782 +0000 UTC m=+2287.797224173" watchObservedRunningTime="2026-10-30 23:45:01.909638354 +0000 UTC m=+2287.797544713"
Oct 30 23:45:01.942188 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:01.942149    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl" podUID="ba44784b-5ea3-4e23-81a4-09e94531aceb" containerName="payload-pre-processing" containerID="cri-o://308ab5f779e59d293c8474379980f14543ac24f520d857b3e9bc7d0326892d18" gracePeriod=25
Oct 30 23:45:02.023652 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.023582    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr" podStartSLOduration=2.023565917 podStartE2EDuration="2.023565917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:02.022327787 +0000 UTC m=+2287.910234142" watchObservedRunningTime="2026-10-30 23:45:02.023565917 +0000 UTC m=+2287.911472271"
Oct 30 23:45:02.135517 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.135458    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79" podStartSLOduration=2.135440072 podStartE2EDuration="2.135440072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:02.133276583 +0000 UTC m=+2288.021182937" watchObservedRunningTime="2026-10-30 23:45:02.135440072 +0000 UTC m=+2288.023346426"
Oct 30 23:45:02.247291 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.247239    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp" podStartSLOduration=2.247223975 podStartE2EDuration="2.247223975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:02.245259528 +0000 UTC m=+2288.133165882" watchObservedRunningTime="2026-10-30 23:45:02.247223975 +0000 UTC m=+2288.135130329"
Oct 30 23:45:02.411361 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.411313    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq" podStartSLOduration=2.411297866 podStartE2EDuration="2.411297866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:02.408285282 +0000 UTC m=+2288.296191633" watchObservedRunningTime="2026-10-30 23:45:02.411297866 +0000 UTC m=+2288.299204219"
Oct 30 23:45:02.574325 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.574195    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn" podStartSLOduration=2.574176719 podStartE2EDuration="2.574176719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:02.573453653 +0000 UTC m=+2288.461360006" watchObservedRunningTime="2026-10-30 23:45:02.574176719 +0000 UTC m=+2288.462083074"
Oct 30 23:45:02.684529 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.684468    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"]
Oct 30 23:45:02.688489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.688462    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-989a4f/distinct-2-w4-989a4f-kserve-7cdd664d4c-s7h4r"]
Oct 30 23:45:02.820381 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.820351    5078 generic.go:356] "Generic (PLEG): container finished" podID="00a24baa-07c5-4252-a4e9-8ca2dda0f829" containerID="f049223af5cd405b205dcfcae074efc659f1ae24507e9121302a64641af0f0e3" exitCode=60
Oct 30 23:45:02.820814 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.820438    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn" event={"ID":"00a24baa-07c5-4252-a4e9-8ca2dda0f829","Type":"ContainerDied","Data":"f049223af5cd405b205dcfcae074efc659f1ae24507e9121302a64641af0f0e3"}
Oct 30 23:45:02.820814 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.820717    5078 scope.go:122] "RemoveContainer" containerID="f049223af5cd405b205dcfcae074efc659f1ae24507e9121302a64641af0f0e3"
Oct 30 23:45:02.822083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.822062    5078 generic.go:356] "Generic (PLEG): container finished" podID="53dc275c-99f8-40a6-ae1e-7acb22c8718c" containerID="5229467a22226be02c389cee0c0c1b60a5b10b32df486f6c0029c7a652f65ca5" exitCode=60
Oct 30 23:45:02.822179 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.822139    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr" event={"ID":"53dc275c-99f8-40a6-ae1e-7acb22c8718c","Type":"ContainerDied","Data":"5229467a22226be02c389cee0c0c1b60a5b10b32df486f6c0029c7a652f65ca5"}
Oct 30 23:45:02.822431 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.822332    5078 scope.go:122] "RemoveContainer" containerID="5229467a22226be02c389cee0c0c1b60a5b10b32df486f6c0029c7a652f65ca5"
Oct 30 23:45:02.823891 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.823868    5078 generic.go:356] "Generic (PLEG): container finished" podID="977fb1cb-c631-476a-8a27-373ce7cace0a" containerID="49a3bda3429f5e03e5f8bfc3055e68d459238f7d16a3aab38bace8fce23c7076" exitCode=60
Oct 30 23:45:02.823996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.823948    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5" event={"ID":"977fb1cb-c631-476a-8a27-373ce7cace0a","Type":"ContainerDied","Data":"49a3bda3429f5e03e5f8bfc3055e68d459238f7d16a3aab38bace8fce23c7076"}
Oct 30 23:45:02.824238 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.824164    5078 scope.go:122] "RemoveContainer" containerID="49a3bda3429f5e03e5f8bfc3055e68d459238f7d16a3aab38bace8fce23c7076"
Oct 30 23:45:02.825582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.825530    5078 generic.go:356] "Generic (PLEG): container finished" podID="ca0cd089-df4b-4d76-854b-217b6c93b42c" containerID="81cf70e9f48c35db0c085f1b48d5b28653d4da4f8c94392e71e98fedee38e71d" exitCode=60
Oct 30 23:45:02.825680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.825624    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq" event={"ID":"ca0cd089-df4b-4d76-854b-217b6c93b42c","Type":"ContainerDied","Data":"81cf70e9f48c35db0c085f1b48d5b28653d4da4f8c94392e71e98fedee38e71d"}
Oct 30 23:45:02.825832 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.825817    5078 scope.go:122] "RemoveContainer" containerID="81cf70e9f48c35db0c085f1b48d5b28653d4da4f8c94392e71e98fedee38e71d"
Oct 30 23:45:02.827337 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.827319    5078 generic.go:356] "Generic (PLEG): container finished" podID="acddcfa4-c6f5-413d-a172-74ec4ab2433f" containerID="a09786b773d0eb9f83a4b006cd73937ccdadecdbcfa3616cdd02d2f1c8a88eb0" exitCode=60
Oct 30 23:45:02.827444 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.827396    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79" event={"ID":"acddcfa4-c6f5-413d-a172-74ec4ab2433f","Type":"ContainerDied","Data":"a09786b773d0eb9f83a4b006cd73937ccdadecdbcfa3616cdd02d2f1c8a88eb0"}
Oct 30 23:45:02.827576 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.827557    5078 scope.go:122] "RemoveContainer" containerID="a09786b773d0eb9f83a4b006cd73937ccdadecdbcfa3616cdd02d2f1c8a88eb0"
Oct 30 23:45:02.828963 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.828946    5078 generic.go:356] "Generic (PLEG): container finished" podID="945e980d-0501-49a4-bad0-4af78c316441" containerID="39edd45749487533bf70f99669f3689fa4df7c3369c61b79ad9b0bbb369090f3" exitCode=60
Oct 30 23:45:02.829058 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.829035    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp" event={"ID":"945e980d-0501-49a4-bad0-4af78c316441","Type":"ContainerDied","Data":"39edd45749487533bf70f99669f3689fa4df7c3369c61b79ad9b0bbb369090f3"}
Oct 30 23:45:02.829174 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.829161    5078 scope.go:122] "RemoveContainer" containerID="39edd45749487533bf70f99669f3689fa4df7c3369c61b79ad9b0bbb369090f3"
Oct 30 23:45:02.832310 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.832296    5078 generic.go:356] "Generic (PLEG): container finished" podID="3349e30c-c66e-4f0f-b640-f0d1d0c50588" containerID="a4c8767653311b14e46927b0e296ebaec0d91b2fc0eb4348e15d14892a0c7d50" exitCode=60
Oct 30 23:45:02.832356 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.832337    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8" event={"ID":"3349e30c-c66e-4f0f-b640-f0d1d0c50588","Type":"ContainerDied","Data":"a4c8767653311b14e46927b0e296ebaec0d91b2fc0eb4348e15d14892a0c7d50"}
Oct 30 23:45:02.832562 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:02.832551    5078 scope.go:122] "RemoveContainer" containerID="a4c8767653311b14e46927b0e296ebaec0d91b2fc0eb4348e15d14892a0c7d50"
Oct 30 23:45:03.015043 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:03.015014    5078 scope.go:122] "RemoveContainer" containerID="8d0c2464b7164e633b97d104922e28c90f8c14c7fe9281b4c88991db2aab5b13"
Oct 30 23:45:03.840435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:03.840399    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8" event={"ID":"3349e30c-c66e-4f0f-b640-f0d1d0c50588","Type":"ContainerStarted","Data":"6e9a403921e7965cbb0adb5c70db0f5df4b1354c8a6651c7786c4b4f078c2647"}
Oct 30 23:45:03.842462 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:03.842433    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn" event={"ID":"00a24baa-07c5-4252-a4e9-8ca2dda0f829","Type":"ContainerStarted","Data":"334e28468fb5b483f2d4fe0c29aebb9eba589008625ba4b13b44dc2a6d2e8789"}
Oct 30 23:45:03.844329 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:03.844308    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr" event={"ID":"53dc275c-99f8-40a6-ae1e-7acb22c8718c","Type":"ContainerStarted","Data":"527d3b43193dee63de075922d6dad97c71e70be95a5b506a9da0a9783ad2dfb1"}
Oct 30 23:45:03.846106 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:03.846087    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5" event={"ID":"977fb1cb-c631-476a-8a27-373ce7cace0a","Type":"ContainerStarted","Data":"ccf0607ecf8ff3b6b8983f33914ef643139487c51082c03f09d2990518328368"}
Oct 30 23:45:03.847930 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:03.847904    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq" event={"ID":"ca0cd089-df4b-4d76-854b-217b6c93b42c","Type":"ContainerStarted","Data":"f99ada36f0083f9394fdfce793682407b9541295b681e02c843b2e37028f490c"}
Oct 30 23:45:03.849714 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:03.849691    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79" event={"ID":"acddcfa4-c6f5-413d-a172-74ec4ab2433f","Type":"ContainerStarted","Data":"3906d26b1fa3ab472439e7caf0b822ff17b5da044cb0474d0ec2d08622f5564d"}
Oct 30 23:45:03.851653 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:03.851632    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp" event={"ID":"945e980d-0501-49a4-bad0-4af78c316441","Type":"ContainerStarted","Data":"75521f7dee8016e8a14c3237c4aa3909ff0ad56b99a430588d0316780ab33763"}
Oct 30 23:45:03.953329 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:03.952891    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs" podUID="b6416daa-de15-47d8-aca5-da1b417b6750" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:45:03.961085 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:03.961012    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl" podUID="ba44784b-5ea3-4e23-81a4-09e94531aceb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:45:04.299181 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.299150    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" path="/var/lib/kubelet/pods/cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d/volumes"
Oct 30 23:45:04.351881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.351855    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82"
Oct 30 23:45:04.354865 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.354845    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm"
Oct 30 23:45:04.517228 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.517197    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fca3a8d-ef11-4d95-ba93-bcafa869e2c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fca3a8d-ef11-4d95-ba93-bcafa869e2c1-plugins-config-volume\") pod \"6fca3a8d-ef11-4d95-ba93-bcafa869e2c1\" (UID: \"6fca3a8d-ef11-4d95-ba93-bcafa869e2c1\") "
Oct 30 23:45:04.517401 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.517276    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b6065b-de7d-41b2-9733-42f56d923d59-kube-api-access-4rd74\" (UniqueName: \"kubernetes.io/projected/b6b6065b-de7d-41b2-9733-42f56d923d59-kube-api-access-4rd74\") pod \"b6b6065b-de7d-41b2-9733-42f56d923d59\" (UID: \"b6b6065b-de7d-41b2-9733-42f56d923d59\") "
Oct 30 23:45:04.517401 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.517329    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fca3a8d-ef11-4d95-ba93-bcafa869e2c1-kube-api-access-w6fmp\" (UniqueName: \"kubernetes.io/projected/6fca3a8d-ef11-4d95-ba93-bcafa869e2c1-kube-api-access-w6fmp\") pod \"6fca3a8d-ef11-4d95-ba93-bcafa869e2c1\" (UID: \"6fca3a8d-ef11-4d95-ba93-bcafa869e2c1\") "
Oct 30 23:45:04.517536 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.517455    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b6065b-de7d-41b2-9733-42f56d923d59-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b6065b-de7d-41b2-9733-42f56d923d59-plugins-config-volume\") pod \"b6b6065b-de7d-41b2-9733-42f56d923d59\" (UID: \"b6b6065b-de7d-41b2-9733-42f56d923d59\") "
Oct 30 23:45:04.517634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.517590    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fca3a8d-ef11-4d95-ba93-bcafa869e2c1-plugins-config-volume" pod "6fca3a8d-ef11-4d95-ba93-bcafa869e2c1" (UID: "6fca3a8d-ef11-4d95-ba93-bcafa869e2c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:04.517859 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.517836    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b6065b-de7d-41b2-9733-42f56d923d59-plugins-config-volume" pod "b6b6065b-de7d-41b2-9733-42f56d923d59" (UID: "b6b6065b-de7d-41b2-9733-42f56d923d59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:04.518042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.517986    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6b6065b-de7d-41b2-9733-42f56d923d59-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:04.518042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.518014    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fca3a8d-ef11-4d95-ba93-bcafa869e2c1-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:04.519575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.519555    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fca3a8d-ef11-4d95-ba93-bcafa869e2c1-kube-api-access-w6fmp" pod "6fca3a8d-ef11-4d95-ba93-bcafa869e2c1" (UID: "6fca3a8d-ef11-4d95-ba93-bcafa869e2c1"). InnerVolumeSpecName "kube-api-access-w6fmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:04.519685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.519627    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b6065b-de7d-41b2-9733-42f56d923d59-kube-api-access-4rd74" pod "b6b6065b-de7d-41b2-9733-42f56d923d59" (UID: "b6b6065b-de7d-41b2-9733-42f56d923d59"). InnerVolumeSpecName "kube-api-access-4rd74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:04.524131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.524088    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5" podStartSLOduration=4.524074335 podStartE2EDuration="4.524074335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:04.520814494 +0000 UTC m=+2290.408720847" watchObservedRunningTime="2026-10-30 23:45:04.524074335 +0000 UTC m=+2290.411980687"
Oct 30 23:45:04.619153 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.619122    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rd74\" (UniqueName: \"kubernetes.io/projected/b6b6065b-de7d-41b2-9733-42f56d923d59-kube-api-access-4rd74\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:04.619153 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.619151    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6fmp\" (UniqueName: \"kubernetes.io/projected/6fca3a8d-ef11-4d95-ba93-bcafa869e2c1-kube-api-access-w6fmp\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:04.862162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.862124    5078 generic.go:356] "Generic (PLEG): container finished" podID="53dc275c-99f8-40a6-ae1e-7acb22c8718c" containerID="527d3b43193dee63de075922d6dad97c71e70be95a5b506a9da0a9783ad2dfb1" exitCode=60
Oct 30 23:45:04.862621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.862237    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr" event={"ID":"53dc275c-99f8-40a6-ae1e-7acb22c8718c","Type":"ContainerDied","Data":"527d3b43193dee63de075922d6dad97c71e70be95a5b506a9da0a9783ad2dfb1"}
Oct 30 23:45:04.862621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.862306    5078 scope.go:122] "RemoveContainer" containerID="5229467a22226be02c389cee0c0c1b60a5b10b32df486f6c0029c7a652f65ca5"
Oct 30 23:45:04.862699 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.862671    5078 scope.go:122] "RemoveContainer" containerID="527d3b43193dee63de075922d6dad97c71e70be95a5b506a9da0a9783ad2dfb1"
Oct 30 23:45:04.862983 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:04.862964    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr_odh-ai-gateway-infra(53dc275c-99f8-40a6-ae1e-7acb22c8718c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr" podUID="53dc275c-99f8-40a6-ae1e-7acb22c8718c"
Oct 30 23:45:04.865870 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.865849    5078 generic.go:356] "Generic (PLEG): container finished" podID="977fb1cb-c631-476a-8a27-373ce7cace0a" containerID="ccf0607ecf8ff3b6b8983f33914ef643139487c51082c03f09d2990518328368" exitCode=60
Oct 30 23:45:04.865976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.865917    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5" event={"ID":"977fb1cb-c631-476a-8a27-373ce7cace0a","Type":"ContainerDied","Data":"ccf0607ecf8ff3b6b8983f33914ef643139487c51082c03f09d2990518328368"}
Oct 30 23:45:04.866142 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.866128    5078 scope.go:122] "RemoveContainer" containerID="ccf0607ecf8ff3b6b8983f33914ef643139487c51082c03f09d2990518328368"
Oct 30 23:45:04.866448 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:04.866429    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5_odh-ai-gateway-infra(977fb1cb-c631-476a-8a27-373ce7cace0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5" podUID="977fb1cb-c631-476a-8a27-373ce7cace0a"
Oct 30 23:45:04.867834 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.867814    5078 generic.go:356] "Generic (PLEG): container finished" podID="ca0cd089-df4b-4d76-854b-217b6c93b42c" containerID="f99ada36f0083f9394fdfce793682407b9541295b681e02c843b2e37028f490c" exitCode=60
Oct 30 23:45:04.867946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.867895    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq" event={"ID":"ca0cd089-df4b-4d76-854b-217b6c93b42c","Type":"ContainerDied","Data":"f99ada36f0083f9394fdfce793682407b9541295b681e02c843b2e37028f490c"}
Oct 30 23:45:04.868113 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.868098    5078 scope.go:122] "RemoveContainer" containerID="f99ada36f0083f9394fdfce793682407b9541295b681e02c843b2e37028f490c"
Oct 30 23:45:04.868376 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:04.868353    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq_odh-ai-gateway-infra(ca0cd089-df4b-4d76-854b-217b6c93b42c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq" podUID="ca0cd089-df4b-4d76-854b-217b6c93b42c"
Oct 30 23:45:04.869927 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.869866    5078 generic.go:356] "Generic (PLEG): container finished" podID="acddcfa4-c6f5-413d-a172-74ec4ab2433f" containerID="3906d26b1fa3ab472439e7caf0b822ff17b5da044cb0474d0ec2d08622f5564d" exitCode=60
Oct 30 23:45:04.870022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.869938    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79" event={"ID":"acddcfa4-c6f5-413d-a172-74ec4ab2433f","Type":"ContainerDied","Data":"3906d26b1fa3ab472439e7caf0b822ff17b5da044cb0474d0ec2d08622f5564d"}
Oct 30 23:45:04.870109 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.870091    5078 scope.go:122] "RemoveContainer" containerID="3906d26b1fa3ab472439e7caf0b822ff17b5da044cb0474d0ec2d08622f5564d"
Oct 30 23:45:04.870368 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:04.870339    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890065-qfd79_odh-ai-gateway-infra(acddcfa4-c6f5-413d-a172-74ec4ab2433f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79" podUID="acddcfa4-c6f5-413d-a172-74ec4ab2433f"
Oct 30 23:45:04.871925 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.871904    5078 generic.go:356] "Generic (PLEG): container finished" podID="945e980d-0501-49a4-bad0-4af78c316441" containerID="75521f7dee8016e8a14c3237c4aa3909ff0ad56b99a430588d0316780ab33763" exitCode=60
Oct 30 23:45:04.872048 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.872027    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp" event={"ID":"945e980d-0501-49a4-bad0-4af78c316441","Type":"ContainerDied","Data":"75521f7dee8016e8a14c3237c4aa3909ff0ad56b99a430588d0316780ab33763"}
Oct 30 23:45:04.872150 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.872137    5078 scope.go:122] "RemoveContainer" containerID="75521f7dee8016e8a14c3237c4aa3909ff0ad56b99a430588d0316780ab33763"
Oct 30 23:45:04.872373 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:04.872356    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp_odh-ai-gateway-infra(945e980d-0501-49a4-bad0-4af78c316441)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp" podUID="945e980d-0501-49a4-bad0-4af78c316441"
Oct 30 23:45:04.873833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.873812    5078 generic.go:356] "Generic (PLEG): container finished" podID="3349e30c-c66e-4f0f-b640-f0d1d0c50588" containerID="6e9a403921e7965cbb0adb5c70db0f5df4b1354c8a6651c7786c4b4f078c2647" exitCode=60
Oct 30 23:45:04.873930 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.873893    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8" event={"ID":"3349e30c-c66e-4f0f-b640-f0d1d0c50588","Type":"ContainerDied","Data":"6e9a403921e7965cbb0adb5c70db0f5df4b1354c8a6651c7786c4b4f078c2647"}
Oct 30 23:45:04.874053 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.874038    5078 scope.go:122] "RemoveContainer" containerID="6e9a403921e7965cbb0adb5c70db0f5df4b1354c8a6651c7786c4b4f078c2647"
Oct 30 23:45:04.874239 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:04.874223    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8_odh-ai-gateway-infra(3349e30c-c66e-4f0f-b640-f0d1d0c50588)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8" podUID="3349e30c-c66e-4f0f-b640-f0d1d0c50588"
Oct 30 23:45:04.875587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.875563    5078 generic.go:356] "Generic (PLEG): container finished" podID="6fca3a8d-ef11-4d95-ba93-bcafa869e2c1" containerID="155b6086942f9dd6951d898325178e8b58cffc1ad7380d5c2779304a03dd83ff" exitCode=0
Oct 30 23:45:04.875587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.875643    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82" event={"ID":"6fca3a8d-ef11-4d95-ba93-bcafa869e2c1","Type":"ContainerDied","Data":"155b6086942f9dd6951d898325178e8b58cffc1ad7380d5c2779304a03dd83ff"}
Oct 30 23:45:04.875784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.875688    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82" event={"ID":"6fca3a8d-ef11-4d95-ba93-bcafa869e2c1","Type":"ContainerDied","Data":"e89d859c729afd8597b72aaee7f40e7ce448ea547c2bf688a4ff75a3b0e76f86"}
Oct 30 23:45:04.875784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.875648    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82"
Oct 30 23:45:04.878001 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.877786    5078 scope.go:122] "RemoveContainer" containerID="49a3bda3429f5e03e5f8bfc3055e68d459238f7d16a3aab38bace8fce23c7076"
Oct 30 23:45:04.878162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.878142    5078 generic.go:356] "Generic (PLEG): container finished" podID="00a24baa-07c5-4252-a4e9-8ca2dda0f829" containerID="334e28468fb5b483f2d4fe0c29aebb9eba589008625ba4b13b44dc2a6d2e8789" exitCode=60
Oct 30 23:45:04.878270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.878220    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn" event={"ID":"00a24baa-07c5-4252-a4e9-8ca2dda0f829","Type":"ContainerDied","Data":"334e28468fb5b483f2d4fe0c29aebb9eba589008625ba4b13b44dc2a6d2e8789"}
Oct 30 23:45:04.878439 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.878418    5078 scope.go:122] "RemoveContainer" containerID="334e28468fb5b483f2d4fe0c29aebb9eba589008625ba4b13b44dc2a6d2e8789"
Oct 30 23:45:04.878707 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:04.878686    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn_odh-ai-gateway-infra(00a24baa-07c5-4252-a4e9-8ca2dda0f829)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn" podUID="00a24baa-07c5-4252-a4e9-8ca2dda0f829"
Oct 30 23:45:04.879518 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.879499    5078 generic.go:356] "Generic (PLEG): container finished" podID="b6b6065b-de7d-41b2-9733-42f56d923d59" containerID="636769bcd33bd31630837964dd6b66d948889f64bb604f3f309b3a3d5d3e66e3" exitCode=0
Oct 30 23:45:04.879632 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.879550    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm"
Oct 30 23:45:04.879632 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.879559    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm" event={"ID":"b6b6065b-de7d-41b2-9733-42f56d923d59","Type":"ContainerDied","Data":"636769bcd33bd31630837964dd6b66d948889f64bb604f3f309b3a3d5d3e66e3"}
Oct 30 23:45:04.879632 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.879586    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm" event={"ID":"b6b6065b-de7d-41b2-9733-42f56d923d59","Type":"ContainerDied","Data":"20b9e27ef41f643a6e18115b2b2edcb4fd726b20efe089e736307a23bf299081"}
Oct 30 23:45:04.892466 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.892448    5078 scope.go:122] "RemoveContainer" containerID="81cf70e9f48c35db0c085f1b48d5b28653d4da4f8c94392e71e98fedee38e71d"
Oct 30 23:45:04.905089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.905073    5078 scope.go:122] "RemoveContainer" containerID="a09786b773d0eb9f83a4b006cd73937ccdadecdbcfa3616cdd02d2f1c8a88eb0"
Oct 30 23:45:04.917484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.917460    5078 scope.go:122] "RemoveContainer" containerID="39edd45749487533bf70f99669f3689fa4df7c3369c61b79ad9b0bbb369090f3"
Oct 30 23:45:04.932080 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.932052    5078 scope.go:122] "RemoveContainer" containerID="a4c8767653311b14e46927b0e296ebaec0d91b2fc0eb4348e15d14892a0c7d50"
Oct 30 23:45:04.946115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.946096    5078 scope.go:122] "RemoveContainer" containerID="155b6086942f9dd6951d898325178e8b58cffc1ad7380d5c2779304a03dd83ff"
Oct 30 23:45:04.955214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.955196    5078 scope.go:122] "RemoveContainer" containerID="155b6086942f9dd6951d898325178e8b58cffc1ad7380d5c2779304a03dd83ff"
Oct 30 23:45:04.955532 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:04.955511    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"155b6086942f9dd6951d898325178e8b58cffc1ad7380d5c2779304a03dd83ff\": container with ID starting with 155b6086942f9dd6951d898325178e8b58cffc1ad7380d5c2779304a03dd83ff not found: ID does not exist" containerID="155b6086942f9dd6951d898325178e8b58cffc1ad7380d5c2779304a03dd83ff"
Oct 30 23:45:04.955628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.955549    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"155b6086942f9dd6951d898325178e8b58cffc1ad7380d5c2779304a03dd83ff"} err="failed to get container status \"155b6086942f9dd6951d898325178e8b58cffc1ad7380d5c2779304a03dd83ff\": rpc error: code = NotFound desc = could not find container \"155b6086942f9dd6951d898325178e8b58cffc1ad7380d5c2779304a03dd83ff\": container with ID starting with 155b6086942f9dd6951d898325178e8b58cffc1ad7380d5c2779304a03dd83ff not found: ID does not exist"
Oct 30 23:45:04.955628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.955568    5078 scope.go:122] "RemoveContainer" containerID="f049223af5cd405b205dcfcae074efc659f1ae24507e9121302a64641af0f0e3"
Oct 30 23:45:04.968144 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.968123    5078 scope.go:122] "RemoveContainer" containerID="636769bcd33bd31630837964dd6b66d948889f64bb604f3f309b3a3d5d3e66e3"
Oct 30 23:45:04.979290 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.979264    5078 scope.go:122] "RemoveContainer" containerID="636769bcd33bd31630837964dd6b66d948889f64bb604f3f309b3a3d5d3e66e3"
Oct 30 23:45:04.979859 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:04.979833    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"636769bcd33bd31630837964dd6b66d948889f64bb604f3f309b3a3d5d3e66e3\": container with ID starting with 636769bcd33bd31630837964dd6b66d948889f64bb604f3f309b3a3d5d3e66e3 not found: ID does not exist" containerID="636769bcd33bd31630837964dd6b66d948889f64bb604f3f309b3a3d5d3e66e3"
Oct 30 23:45:04.979946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:04.979871    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"636769bcd33bd31630837964dd6b66d948889f64bb604f3f309b3a3d5d3e66e3"} err="failed to get container status \"636769bcd33bd31630837964dd6b66d948889f64bb604f3f309b3a3d5d3e66e3\": rpc error: code = NotFound desc = could not find container \"636769bcd33bd31630837964dd6b66d948889f64bb604f3f309b3a3d5d3e66e3\": container with ID starting with 636769bcd33bd31630837964dd6b66d948889f64bb604f3f309b3a3d5d3e66e3 not found: ID does not exist"
Oct 30 23:45:05.151863 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:05.151766    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm"]
Oct 30 23:45:05.154928 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:05.154902    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-057303-599bdbd7c9-jd7cm"]
Oct 30 23:45:05.375250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:05.375206    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82"]
Oct 30 23:45:05.378314 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:05.378289    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-057303-544cd447b7-j8p82"]
Oct 30 23:45:05.888821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:05.888791    5078 scope.go:122] "RemoveContainer" containerID="6e9a403921e7965cbb0adb5c70db0f5df4b1354c8a6651c7786c4b4f078c2647"
Oct 30 23:45:05.889302 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:05.889106    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8_odh-ai-gateway-infra(3349e30c-c66e-4f0f-b640-f0d1d0c50588)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8" podUID="3349e30c-c66e-4f0f-b640-f0d1d0c50588"
Oct 30 23:45:05.891818 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:05.891796    5078 scope.go:122] "RemoveContainer" containerID="334e28468fb5b483f2d4fe0c29aebb9eba589008625ba4b13b44dc2a6d2e8789"
Oct 30 23:45:05.892080 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:05.892060    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn_odh-ai-gateway-infra(00a24baa-07c5-4252-a4e9-8ca2dda0f829)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn" podUID="00a24baa-07c5-4252-a4e9-8ca2dda0f829"
Oct 30 23:45:05.894722 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:05.894703    5078 scope.go:122] "RemoveContainer" containerID="527d3b43193dee63de075922d6dad97c71e70be95a5b506a9da0a9783ad2dfb1"
Oct 30 23:45:05.895004 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:05.894980    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr_odh-ai-gateway-infra(53dc275c-99f8-40a6-ae1e-7acb22c8718c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr" podUID="53dc275c-99f8-40a6-ae1e-7acb22c8718c"
Oct 30 23:45:05.896263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:05.896246    5078 scope.go:122] "RemoveContainer" containerID="ccf0607ecf8ff3b6b8983f33914ef643139487c51082c03f09d2990518328368"
Oct 30 23:45:05.896516 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:05.896495    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5_odh-ai-gateway-infra(977fb1cb-c631-476a-8a27-373ce7cace0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5" podUID="977fb1cb-c631-476a-8a27-373ce7cace0a"
Oct 30 23:45:05.897751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:05.897734    5078 scope.go:122] "RemoveContainer" containerID="f99ada36f0083f9394fdfce793682407b9541295b681e02c843b2e37028f490c"
Oct 30 23:45:05.897985 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:05.897965    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq_odh-ai-gateway-infra(ca0cd089-df4b-4d76-854b-217b6c93b42c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq" podUID="ca0cd089-df4b-4d76-854b-217b6c93b42c"
Oct 30 23:45:05.899501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:05.899482    5078 scope.go:122] "RemoveContainer" containerID="3906d26b1fa3ab472439e7caf0b822ff17b5da044cb0474d0ec2d08622f5564d"
Oct 30 23:45:05.899729 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:05.899701    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890065-qfd79_odh-ai-gateway-infra(acddcfa4-c6f5-413d-a172-74ec4ab2433f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79" podUID="acddcfa4-c6f5-413d-a172-74ec4ab2433f"
Oct 30 23:45:05.901017 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:05.901001    5078 scope.go:122] "RemoveContainer" containerID="75521f7dee8016e8a14c3237c4aa3909ff0ad56b99a430588d0316780ab33763"
Oct 30 23:45:05.901231 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:05.901207    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp_odh-ai-gateway-infra(945e980d-0501-49a4-bad0-4af78c316441)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp" podUID="945e980d-0501-49a4-bad0-4af78c316441"
Oct 30 23:45:06.299557 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:06.299517    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fca3a8d-ef11-4d95-ba93-bcafa869e2c1" path="/var/lib/kubelet/pods/6fca3a8d-ef11-4d95-ba93-bcafa869e2c1/volumes"
Oct 30 23:45:06.300017 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:06.300003    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b6065b-de7d-41b2-9733-42f56d923d59" path="/var/lib/kubelet/pods/b6b6065b-de7d-41b2-9733-42f56d923d59/volumes"
Oct 30 23:45:13.951507 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:13.951469    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs" podUID="b6416daa-de15-47d8-aca5-da1b417b6750" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:45:13.951991 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:13.951947    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl" podUID="ba44784b-5ea3-4e23-81a4-09e94531aceb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:45:16.291388 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:16.291354    5078 scope.go:122] "RemoveContainer" containerID="3906d26b1fa3ab472439e7caf0b822ff17b5da044cb0474d0ec2d08622f5564d"
Oct 30 23:45:16.291862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:16.291568    5078 scope.go:122] "RemoveContainer" containerID="f99ada36f0083f9394fdfce793682407b9541295b681e02c843b2e37028f490c"
Oct 30 23:45:16.976867 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:16.976782    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq" event={"ID":"ca0cd089-df4b-4d76-854b-217b6c93b42c","Type":"ContainerStarted","Data":"bec3fd82fd2f40d7641812c253d8929dd22d2391c661310039445545d92a79d5"}
Oct 30 23:45:16.979015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:16.978981    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79" event={"ID":"acddcfa4-c6f5-413d-a172-74ec4ab2433f","Type":"ContainerStarted","Data":"d94ea5173d5f34c71006ce4f1ab44029a19a089462a24e6c768995eb50d04949"}
Oct 30 23:45:17.297694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:17.291354    5078 scope.go:122] "RemoveContainer" containerID="ccf0607ecf8ff3b6b8983f33914ef643139487c51082c03f09d2990518328368"
Oct 30 23:45:17.297694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:17.296383    5078 scope.go:122] "RemoveContainer" containerID="527d3b43193dee63de075922d6dad97c71e70be95a5b506a9da0a9783ad2dfb1"
Oct 30 23:45:17.306289 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:17.302256    5078 scope.go:122] "RemoveContainer" containerID="334e28468fb5b483f2d4fe0c29aebb9eba589008625ba4b13b44dc2a6d2e8789"
Oct 30 23:45:17.592793 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:17.592740    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:45:17.987111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:17.987079    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn" event={"ID":"00a24baa-07c5-4252-a4e9-8ca2dda0f829","Type":"ContainerStarted","Data":"b14aa756c262c2d1d5a96ee9bec31a8639a2dfcc333c9775cfd9c2adb99ce589"}
Oct 30 23:45:17.989045 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:17.989021    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr" event={"ID":"53dc275c-99f8-40a6-ae1e-7acb22c8718c","Type":"ContainerStarted","Data":"0def2e9d325b26d1da243526600366601841e6dd148a95541a3b6d874448a4cf"}
Oct 30 23:45:17.991038 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:17.991018    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5" event={"ID":"977fb1cb-c631-476a-8a27-373ce7cace0a","Type":"ContainerStarted","Data":"686f35647458e758c36d05fc44ca91da6cbf7df2d2a5345af5dce05e4c68b108"}
Oct 30 23:45:17.992927 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:17.992906    5078 generic.go:356] "Generic (PLEG): container finished" podID="ca0cd089-df4b-4d76-854b-217b6c93b42c" containerID="bec3fd82fd2f40d7641812c253d8929dd22d2391c661310039445545d92a79d5" exitCode=60
Oct 30 23:45:17.993020 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:17.992979    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq" event={"ID":"ca0cd089-df4b-4d76-854b-217b6c93b42c","Type":"ContainerDied","Data":"bec3fd82fd2f40d7641812c253d8929dd22d2391c661310039445545d92a79d5"}
Oct 30 23:45:17.993094 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:17.993026    5078 scope.go:122] "RemoveContainer" containerID="f99ada36f0083f9394fdfce793682407b9541295b681e02c843b2e37028f490c"
Oct 30 23:45:17.993292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:17.993278    5078 scope.go:122] "RemoveContainer" containerID="bec3fd82fd2f40d7641812c253d8929dd22d2391c661310039445545d92a79d5"
Oct 30 23:45:17.993553 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:17.993530    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq_odh-ai-gateway-infra(ca0cd089-df4b-4d76-854b-217b6c93b42c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq" podUID="ca0cd089-df4b-4d76-854b-217b6c93b42c"
Oct 30 23:45:17.995018 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:17.994999    5078 generic.go:356] "Generic (PLEG): container finished" podID="acddcfa4-c6f5-413d-a172-74ec4ab2433f" containerID="d94ea5173d5f34c71006ce4f1ab44029a19a089462a24e6c768995eb50d04949" exitCode=60
Oct 30 23:45:17.995096 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:17.995044    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79" event={"ID":"acddcfa4-c6f5-413d-a172-74ec4ab2433f","Type":"ContainerDied","Data":"d94ea5173d5f34c71006ce4f1ab44029a19a089462a24e6c768995eb50d04949"}
Oct 30 23:45:17.995317 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:17.995304    5078 scope.go:122] "RemoveContainer" containerID="d94ea5173d5f34c71006ce4f1ab44029a19a089462a24e6c768995eb50d04949"
Oct 30 23:45:17.995505 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:17.995490    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890065-qfd79_odh-ai-gateway-infra(acddcfa4-c6f5-413d-a172-74ec4ab2433f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79" podUID="acddcfa4-c6f5-413d-a172-74ec4ab2433f"
Oct 30 23:45:18.006089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:18.006059    5078 scope.go:122] "RemoveContainer" containerID="3906d26b1fa3ab472439e7caf0b822ff17b5da044cb0474d0ec2d08622f5564d"
Oct 30 23:45:18.106802 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:18.106768    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq"]
Oct 30 23:45:18.217779 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:18.217755    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79"]
Oct 30 23:45:18.292071 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:18.292033    5078 scope.go:122] "RemoveContainer" containerID="75521f7dee8016e8a14c3237c4aa3909ff0ad56b99a430588d0316780ab33763"
Oct 30 23:45:18.292491 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:18.292467    5078 scope.go:122] "RemoveContainer" containerID="6e9a403921e7965cbb0adb5c70db0f5df4b1354c8a6651c7786c4b4f078c2647"
Oct 30 23:45:19.003216 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.003131    5078 generic.go:356] "Generic (PLEG): container finished" podID="00a24baa-07c5-4252-a4e9-8ca2dda0f829" containerID="b14aa756c262c2d1d5a96ee9bec31a8639a2dfcc333c9775cfd9c2adb99ce589" exitCode=60
Oct 30 23:45:19.003716 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.003211    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn" event={"ID":"00a24baa-07c5-4252-a4e9-8ca2dda0f829","Type":"ContainerDied","Data":"b14aa756c262c2d1d5a96ee9bec31a8639a2dfcc333c9775cfd9c2adb99ce589"}
Oct 30 23:45:19.003716 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.003286    5078 scope.go:122] "RemoveContainer" containerID="334e28468fb5b483f2d4fe0c29aebb9eba589008625ba4b13b44dc2a6d2e8789"
Oct 30 23:45:19.003844 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.003720    5078 scope.go:122] "RemoveContainer" containerID="b14aa756c262c2d1d5a96ee9bec31a8639a2dfcc333c9775cfd9c2adb99ce589"
Oct 30 23:45:19.004070 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:19.004047    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn_odh-ai-gateway-infra(00a24baa-07c5-4252-a4e9-8ca2dda0f829)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn" podUID="00a24baa-07c5-4252-a4e9-8ca2dda0f829"
Oct 30 23:45:19.005267 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.005248    5078 generic.go:356] "Generic (PLEG): container finished" podID="53dc275c-99f8-40a6-ae1e-7acb22c8718c" containerID="0def2e9d325b26d1da243526600366601841e6dd148a95541a3b6d874448a4cf" exitCode=60
Oct 30 23:45:19.005372 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.005319    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr" event={"ID":"53dc275c-99f8-40a6-ae1e-7acb22c8718c","Type":"ContainerDied","Data":"0def2e9d325b26d1da243526600366601841e6dd148a95541a3b6d874448a4cf"}
Oct 30 23:45:19.005713 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.005695    5078 scope.go:122] "RemoveContainer" containerID="0def2e9d325b26d1da243526600366601841e6dd148a95541a3b6d874448a4cf"
Oct 30 23:45:19.005989 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:19.005969    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr_odh-ai-gateway-infra(53dc275c-99f8-40a6-ae1e-7acb22c8718c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr" podUID="53dc275c-99f8-40a6-ae1e-7acb22c8718c"
Oct 30 23:45:19.007041 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.007021    5078 generic.go:356] "Generic (PLEG): container finished" podID="977fb1cb-c631-476a-8a27-373ce7cace0a" containerID="686f35647458e758c36d05fc44ca91da6cbf7df2d2a5345af5dce05e4c68b108" exitCode=60
Oct 30 23:45:19.007130 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.007091    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5" event={"ID":"977fb1cb-c631-476a-8a27-373ce7cace0a","Type":"ContainerDied","Data":"686f35647458e758c36d05fc44ca91da6cbf7df2d2a5345af5dce05e4c68b108"}
Oct 30 23:45:19.007413 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.007398    5078 scope.go:122] "RemoveContainer" containerID="686f35647458e758c36d05fc44ca91da6cbf7df2d2a5345af5dce05e4c68b108"
Oct 30 23:45:19.007663 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:19.007642    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5_odh-ai-gateway-infra(977fb1cb-c631-476a-8a27-373ce7cace0a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5" podUID="977fb1cb-c631-476a-8a27-373ce7cace0a"
Oct 30 23:45:19.013456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.013424    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp" event={"ID":"945e980d-0501-49a4-bad0-4af78c316441","Type":"ContainerStarted","Data":"9752c2bcb3915e92e338b31b835cf9772e479723795f8ec950a5c5fd73b56d87"}
Oct 30 23:45:19.015669 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.015651    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8" event={"ID":"3349e30c-c66e-4f0f-b640-f0d1d0c50588","Type":"ContainerStarted","Data":"52560103f0915524f19b0338c9ef584ca8234d484a7203be04933ec31738ba5d"}
Oct 30 23:45:19.019270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.019247    5078 scope.go:122] "RemoveContainer" containerID="527d3b43193dee63de075922d6dad97c71e70be95a5b506a9da0a9783ad2dfb1"
Oct 30 23:45:19.034424 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.034391    5078 scope.go:122] "RemoveContainer" containerID="ccf0607ecf8ff3b6b8983f33914ef643139487c51082c03f09d2990518328368"
Oct 30 23:45:19.114091 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.114059    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn"]
Oct 30 23:45:19.183180 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.183153    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq"
Oct 30 23:45:19.204272 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.204250    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79"
Oct 30 23:45:19.272022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.271951    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca0cd089-df4b-4d76-854b-217b6c93b42c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0cd089-df4b-4d76-854b-217b6c93b42c-maas-api-service-ca\") pod \"ca0cd089-df4b-4d76-854b-217b6c93b42c\" (UID: \"ca0cd089-df4b-4d76-854b-217b6c93b42c\") "
Oct 30 23:45:19.272022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.271991    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca0cd089-df4b-4d76-854b-217b6c93b42c-kube-api-access-5qrbd\" (UniqueName: \"kubernetes.io/projected/ca0cd089-df4b-4d76-854b-217b6c93b42c-kube-api-access-5qrbd\") pod \"ca0cd089-df4b-4d76-854b-217b6c93b42c\" (UID: \"ca0cd089-df4b-4d76-854b-217b6c93b42c\") "
Oct 30 23:45:19.272360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.272338    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca0cd089-df4b-4d76-854b-217b6c93b42c-maas-api-service-ca" pod "ca0cd089-df4b-4d76-854b-217b6c93b42c" (UID: "ca0cd089-df4b-4d76-854b-217b6c93b42c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:19.272433 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.272417    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca0cd089-df4b-4d76-854b-217b6c93b42c-maas-api-service-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:19.274368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.274343    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca0cd089-df4b-4d76-854b-217b6c93b42c-kube-api-access-5qrbd" pod "ca0cd089-df4b-4d76-854b-217b6c93b42c" (UID: "ca0cd089-df4b-4d76-854b-217b6c93b42c"). InnerVolumeSpecName "kube-api-access-5qrbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:19.334888 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.334856    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5"]
Oct 30 23:45:19.373834 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.373804    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acddcfa4-c6f5-413d-a172-74ec4ab2433f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acddcfa4-c6f5-413d-a172-74ec4ab2433f-maas-api-service-ca\") pod \"acddcfa4-c6f5-413d-a172-74ec4ab2433f\" (UID: \"acddcfa4-c6f5-413d-a172-74ec4ab2433f\") "
Oct 30 23:45:19.373950 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.373869    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acddcfa4-c6f5-413d-a172-74ec4ab2433f-kube-api-access-4wts2\" (UniqueName: \"kubernetes.io/projected/acddcfa4-c6f5-413d-a172-74ec4ab2433f-kube-api-access-4wts2\") pod \"acddcfa4-c6f5-413d-a172-74ec4ab2433f\" (UID: \"acddcfa4-c6f5-413d-a172-74ec4ab2433f\") "
Oct 30 23:45:19.374194 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.374167    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acddcfa4-c6f5-413d-a172-74ec4ab2433f-maas-api-service-ca" pod "acddcfa4-c6f5-413d-a172-74ec4ab2433f" (UID: "acddcfa4-c6f5-413d-a172-74ec4ab2433f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:19.374629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.374585    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qrbd\" (UniqueName: \"kubernetes.io/projected/ca0cd089-df4b-4d76-854b-217b6c93b42c-kube-api-access-5qrbd\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:19.374754 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.374634    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acddcfa4-c6f5-413d-a172-74ec4ab2433f-maas-api-service-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:19.375972 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.375949    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acddcfa4-c6f5-413d-a172-74ec4ab2433f-kube-api-access-4wts2" pod "acddcfa4-c6f5-413d-a172-74ec4ab2433f" (UID: "acddcfa4-c6f5-413d-a172-74ec4ab2433f"). InnerVolumeSpecName "kube-api-access-4wts2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:19.446447 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.446417    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr"]
Oct 30 23:45:19.476017 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:19.475990    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wts2\" (UniqueName: \"kubernetes.io/projected/acddcfa4-c6f5-413d-a172-74ec4ab2433f-kube-api-access-4wts2\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:20.028164 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.028141    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq"
Oct 30 23:45:20.028561 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.028140    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq" event={"ID":"ca0cd089-df4b-4d76-854b-217b6c93b42c","Type":"ContainerDied","Data":"d1f134832f0b2d3ee3e212d9dfc33246d25cd8852b3eaf3f48512ef707b60d16"}
Oct 30 23:45:20.028561 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.028289    5078 scope.go:122] "RemoveContainer" containerID="bec3fd82fd2f40d7641812c253d8929dd22d2391c661310039445545d92a79d5"
Oct 30 23:45:20.029784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.029762    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79"
Oct 30 23:45:20.029927 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.029897    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79" event={"ID":"acddcfa4-c6f5-413d-a172-74ec4ab2433f","Type":"ContainerDied","Data":"fc55b089975cc379e03521fb70dbe490d9fcf6eb6c89e9ac98d6b5e2c11a39c6"}
Oct 30 23:45:20.032294 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.032266    5078 generic.go:356] "Generic (PLEG): container finished" podID="945e980d-0501-49a4-bad0-4af78c316441" containerID="9752c2bcb3915e92e338b31b835cf9772e479723795f8ec950a5c5fd73b56d87" exitCode=60
Oct 30 23:45:20.032417 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.032303    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp" event={"ID":"945e980d-0501-49a4-bad0-4af78c316441","Type":"ContainerDied","Data":"9752c2bcb3915e92e338b31b835cf9772e479723795f8ec950a5c5fd73b56d87"}
Oct 30 23:45:20.032744 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.032725    5078 scope.go:122] "RemoveContainer" containerID="9752c2bcb3915e92e338b31b835cf9772e479723795f8ec950a5c5fd73b56d87"
Oct 30 23:45:20.033037 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:20.033012    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp_odh-ai-gateway-infra(945e980d-0501-49a4-bad0-4af78c316441)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp" podUID="945e980d-0501-49a4-bad0-4af78c316441"
Oct 30 23:45:20.035199 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.035177    5078 generic.go:356] "Generic (PLEG): container finished" podID="3349e30c-c66e-4f0f-b640-f0d1d0c50588" containerID="52560103f0915524f19b0338c9ef584ca8234d484a7203be04933ec31738ba5d" exitCode=60
Oct 30 23:45:20.035319 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.035279    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8" event={"ID":"3349e30c-c66e-4f0f-b640-f0d1d0c50588","Type":"ContainerDied","Data":"52560103f0915524f19b0338c9ef584ca8234d484a7203be04933ec31738ba5d"}
Oct 30 23:45:20.035731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.035706    5078 scope.go:122] "RemoveContainer" containerID="52560103f0915524f19b0338c9ef584ca8234d484a7203be04933ec31738ba5d"
Oct 30 23:45:20.036018 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:20.035993    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8_odh-ai-gateway-infra(3349e30c-c66e-4f0f-b640-f0d1d0c50588)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8" podUID="3349e30c-c66e-4f0f-b640-f0d1d0c50588"
Oct 30 23:45:20.047169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.047131    5078 scope.go:122] "RemoveContainer" containerID="d94ea5173d5f34c71006ce4f1ab44029a19a089462a24e6c768995eb50d04949"
Oct 30 23:45:20.074869 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.074837    5078 scope.go:122] "RemoveContainer" containerID="75521f7dee8016e8a14c3237c4aa3909ff0ad56b99a430588d0316780ab33763"
Oct 30 23:45:20.094152 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.094049    5078 scope.go:122] "RemoveContainer" containerID="6e9a403921e7965cbb0adb5c70db0f5df4b1354c8a6651c7786c4b4f078c2647"
Oct 30 23:45:20.249143 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.249110    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr"
Oct 30 23:45:20.289325 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.289297    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5"
Oct 30 23:45:20.289474 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.289455    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79"]
Oct 30 23:45:20.292765 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.292745    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn"
Oct 30 23:45:20.304627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.304589    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890065-qfd79"]
Oct 30 23:45:20.390748 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.390720    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53dc275c-99f8-40a6-ae1e-7acb22c8718c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53dc275c-99f8-40a6-ae1e-7acb22c8718c-maas-api-service-ca\") pod \"53dc275c-99f8-40a6-ae1e-7acb22c8718c\" (UID: \"53dc275c-99f8-40a6-ae1e-7acb22c8718c\") "
Oct 30 23:45:20.390920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.390754    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/977fb1cb-c631-476a-8a27-373ce7cace0a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977fb1cb-c631-476a-8a27-373ce7cace0a-maas-api-service-ca\") pod \"977fb1cb-c631-476a-8a27-373ce7cace0a\" (UID: \"977fb1cb-c631-476a-8a27-373ce7cace0a\") "
Oct 30 23:45:20.390920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.390807    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/977fb1cb-c631-476a-8a27-373ce7cace0a-kube-api-access-mm8mn\" (UniqueName: \"kubernetes.io/projected/977fb1cb-c631-476a-8a27-373ce7cace0a-kube-api-access-mm8mn\") pod \"977fb1cb-c631-476a-8a27-373ce7cace0a\" (UID: \"977fb1cb-c631-476a-8a27-373ce7cace0a\") "
Oct 30 23:45:20.390920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.390864    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a24baa-07c5-4252-a4e9-8ca2dda0f829-kube-api-access-9htrv\" (UniqueName: \"kubernetes.io/projected/00a24baa-07c5-4252-a4e9-8ca2dda0f829-kube-api-access-9htrv\") pod \"00a24baa-07c5-4252-a4e9-8ca2dda0f829\" (UID: \"00a24baa-07c5-4252-a4e9-8ca2dda0f829\") "
Oct 30 23:45:20.391105 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.391019    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a24baa-07c5-4252-a4e9-8ca2dda0f829-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a24baa-07c5-4252-a4e9-8ca2dda0f829-maas-api-service-ca\") pod \"00a24baa-07c5-4252-a4e9-8ca2dda0f829\" (UID: \"00a24baa-07c5-4252-a4e9-8ca2dda0f829\") "
Oct 30 23:45:20.391105 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.391051    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53dc275c-99f8-40a6-ae1e-7acb22c8718c-kube-api-access-hbblr\" (UniqueName: \"kubernetes.io/projected/53dc275c-99f8-40a6-ae1e-7acb22c8718c-kube-api-access-hbblr\") pod \"53dc275c-99f8-40a6-ae1e-7acb22c8718c\" (UID: \"53dc275c-99f8-40a6-ae1e-7acb22c8718c\") "
Oct 30 23:45:20.391225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.391114    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53dc275c-99f8-40a6-ae1e-7acb22c8718c-maas-api-service-ca" pod "53dc275c-99f8-40a6-ae1e-7acb22c8718c" (UID: "53dc275c-99f8-40a6-ae1e-7acb22c8718c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:20.391306 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.391202    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/977fb1cb-c631-476a-8a27-373ce7cace0a-maas-api-service-ca" pod "977fb1cb-c631-476a-8a27-373ce7cace0a" (UID: "977fb1cb-c631-476a-8a27-373ce7cace0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:20.391433 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.391372    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a24baa-07c5-4252-a4e9-8ca2dda0f829-maas-api-service-ca" pod "00a24baa-07c5-4252-a4e9-8ca2dda0f829" (UID: "00a24baa-07c5-4252-a4e9-8ca2dda0f829"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:20.391951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.391932    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a24baa-07c5-4252-a4e9-8ca2dda0f829-maas-api-service-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:20.392042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.391954    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53dc275c-99f8-40a6-ae1e-7acb22c8718c-maas-api-service-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:20.392042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.391968    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977fb1cb-c631-476a-8a27-373ce7cace0a-maas-api-service-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:20.393263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.393234    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977fb1cb-c631-476a-8a27-373ce7cace0a-kube-api-access-mm8mn" pod "977fb1cb-c631-476a-8a27-373ce7cace0a" (UID: "977fb1cb-c631-476a-8a27-373ce7cace0a"). InnerVolumeSpecName "kube-api-access-mm8mn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:20.393850 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.393813    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a24baa-07c5-4252-a4e9-8ca2dda0f829-kube-api-access-9htrv" pod "00a24baa-07c5-4252-a4e9-8ca2dda0f829" (UID: "00a24baa-07c5-4252-a4e9-8ca2dda0f829"). InnerVolumeSpecName "kube-api-access-9htrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:20.393941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.393851    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53dc275c-99f8-40a6-ae1e-7acb22c8718c-kube-api-access-hbblr" pod "53dc275c-99f8-40a6-ae1e-7acb22c8718c" (UID: "53dc275c-99f8-40a6-ae1e-7acb22c8718c"). InnerVolumeSpecName "kube-api-access-hbblr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:20.493621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.493570    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm8mn\" (UniqueName: \"kubernetes.io/projected/977fb1cb-c631-476a-8a27-373ce7cace0a-kube-api-access-mm8mn\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:20.493751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.493635    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9htrv\" (UniqueName: \"kubernetes.io/projected/00a24baa-07c5-4252-a4e9-8ca2dda0f829-kube-api-access-9htrv\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:20.493751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.493651    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbblr\" (UniqueName: \"kubernetes.io/projected/53dc275c-99f8-40a6-ae1e-7acb22c8718c-kube-api-access-hbblr\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:20.494283 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.494263    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8"]
Oct 30 23:45:20.518736 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.518706    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq"]
Oct 30 23:45:20.524828 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.524807    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-057303-29890065-g84rq"]
Oct 30 23:45:20.834781 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:20.834746    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp"]
Oct 30 23:45:21.050150 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.050126    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn"
Oct 30 23:45:21.050553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.050158    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn" event={"ID":"00a24baa-07c5-4252-a4e9-8ca2dda0f829","Type":"ContainerDied","Data":"c37b04094aa4e9ed0e90d014ef2dc25487429ad02d7062084bb0c1c1b5daf1d1"}
Oct 30 23:45:21.050553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.050273    5078 scope.go:122] "RemoveContainer" containerID="b14aa756c262c2d1d5a96ee9bec31a8639a2dfcc333c9775cfd9c2adb99ce589"
Oct 30 23:45:21.051805 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.051786    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr"
Oct 30 23:45:21.051805 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.051793    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr" event={"ID":"53dc275c-99f8-40a6-ae1e-7acb22c8718c","Type":"ContainerDied","Data":"88e63f92592247aaf1f1ea69b726904a5991f249da8351b153d04b4935a41dae"}
Oct 30 23:45:21.053382 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.053360    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5"
Oct 30 23:45:21.053382 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.053370    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5" event={"ID":"977fb1cb-c631-476a-8a27-373ce7cace0a","Type":"ContainerDied","Data":"24c25302a45c93194fe3f921906144fc1d16ce3133468cbbc834f8c68f975383"}
Oct 30 23:45:21.064545 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.064321    5078 scope.go:122] "RemoveContainer" containerID="0def2e9d325b26d1da243526600366601841e6dd148a95541a3b6d874448a4cf"
Oct 30 23:45:21.083673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.083648    5078 scope.go:122] "RemoveContainer" containerID="686f35647458e758c36d05fc44ca91da6cbf7df2d2a5345af5dce05e4c68b108"
Oct 30 23:45:21.260336 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.260316    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp"
Oct 30 23:45:21.262483 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.262466    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8"
Oct 30 23:45:21.320979 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.320953    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr"]
Oct 30 23:45:21.339153 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.339095    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-5dc3c0-29890065-rvtjr"]
Oct 30 23:45:21.409105 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.409071    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/945e980d-0501-49a4-bad0-4af78c316441-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/945e980d-0501-49a4-bad0-4af78c316441-maas-api-service-ca\") pod \"945e980d-0501-49a4-bad0-4af78c316441\" (UID: \"945e980d-0501-49a4-bad0-4af78c316441\") "
Oct 30 23:45:21.409320 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.409160    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3349e30c-c66e-4f0f-b640-f0d1d0c50588-kube-api-access-mnj97\" (UniqueName: \"kubernetes.io/projected/3349e30c-c66e-4f0f-b640-f0d1d0c50588-kube-api-access-mnj97\") pod \"3349e30c-c66e-4f0f-b640-f0d1d0c50588\" (UID: \"3349e30c-c66e-4f0f-b640-f0d1d0c50588\") "
Oct 30 23:45:21.409320 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.409194    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945e980d-0501-49a4-bad0-4af78c316441-kube-api-access-hv8xm\" (UniqueName: \"kubernetes.io/projected/945e980d-0501-49a4-bad0-4af78c316441-kube-api-access-hv8xm\") pod \"945e980d-0501-49a4-bad0-4af78c316441\" (UID: \"945e980d-0501-49a4-bad0-4af78c316441\") "
Oct 30 23:45:21.409320 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.409234    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3349e30c-c66e-4f0f-b640-f0d1d0c50588-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3349e30c-c66e-4f0f-b640-f0d1d0c50588-maas-api-service-ca\") pod \"3349e30c-c66e-4f0f-b640-f0d1d0c50588\" (UID: \"3349e30c-c66e-4f0f-b640-f0d1d0c50588\") "
Oct 30 23:45:21.409568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.409477    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945e980d-0501-49a4-bad0-4af78c316441-maas-api-service-ca" pod "945e980d-0501-49a4-bad0-4af78c316441" (UID: "945e980d-0501-49a4-bad0-4af78c316441"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:21.409689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.409665    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3349e30c-c66e-4f0f-b640-f0d1d0c50588-maas-api-service-ca" pod "3349e30c-c66e-4f0f-b640-f0d1d0c50588" (UID: "3349e30c-c66e-4f0f-b640-f0d1d0c50588"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:21.411413 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.411384    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945e980d-0501-49a4-bad0-4af78c316441-kube-api-access-hv8xm" pod "945e980d-0501-49a4-bad0-4af78c316441" (UID: "945e980d-0501-49a4-bad0-4af78c316441"). InnerVolumeSpecName "kube-api-access-hv8xm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:21.411508 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.411488    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3349e30c-c66e-4f0f-b640-f0d1d0c50588-kube-api-access-mnj97" pod "3349e30c-c66e-4f0f-b640-f0d1d0c50588" (UID: "3349e30c-c66e-4f0f-b640-f0d1d0c50588"). InnerVolumeSpecName "kube-api-access-mnj97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:21.511156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.511125    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/945e980d-0501-49a4-bad0-4af78c316441-maas-api-service-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:21.511156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.511152    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnj97\" (UniqueName: \"kubernetes.io/projected/3349e30c-c66e-4f0f-b640-f0d1d0c50588-kube-api-access-mnj97\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:21.511156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.511161    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hv8xm\" (UniqueName: \"kubernetes.io/projected/945e980d-0501-49a4-bad0-4af78c316441-kube-api-access-hv8xm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:21.511394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.511172    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3349e30c-c66e-4f0f-b640-f0d1d0c50588-maas-api-service-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:21.584698 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.584661    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5"]
Oct 30 23:45:21.589323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.589241    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-26ce6b-29890065-hmsl5"]
Oct 30 23:45:21.686717 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.686673    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn"]
Oct 30 23:45:21.692666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:21.692638    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-aadaa6-29890065-p9jtn"]
Oct 30 23:45:22.070006 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.069977    5078 generic.go:356] "Generic (PLEG): container finished" podID="b6416daa-de15-47d8-aca5-da1b417b6750" containerID="1fefbc4127ab01f1682ca43b2837afd38f510d94b5135c29b32e843072e0851d" exitCode=0
Oct 30 23:45:22.070392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.070078    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs" event={"ID":"b6416daa-de15-47d8-aca5-da1b417b6750","Type":"ContainerDied","Data":"1fefbc4127ab01f1682ca43b2837afd38f510d94b5135c29b32e843072e0851d"}
Oct 30 23:45:22.073003 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.072969    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8" event={"ID":"3349e30c-c66e-4f0f-b640-f0d1d0c50588","Type":"ContainerDied","Data":"c686e6ba6750517db9b457c378010444b9913cbb57852155324c5ce43067f4a9"}
Oct 30 23:45:22.073121 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.073058    5078 scope.go:122] "RemoveContainer" containerID="52560103f0915524f19b0338c9ef584ca8234d484a7203be04933ec31738ba5d"
Oct 30 23:45:22.073241 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.073225    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8"
Oct 30 23:45:22.086799 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.086773    5078 generic.go:356] "Generic (PLEG): container finished" podID="ba44784b-5ea3-4e23-81a4-09e94531aceb" containerID="308ab5f779e59d293c8474379980f14543ac24f520d857b3e9bc7d0326892d18" exitCode=0
Oct 30 23:45:22.086941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.086858    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl" event={"ID":"ba44784b-5ea3-4e23-81a4-09e94531aceb","Type":"ContainerDied","Data":"308ab5f779e59d293c8474379980f14543ac24f520d857b3e9bc7d0326892d18"}
Oct 30 23:45:22.089902 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.089876    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp" event={"ID":"945e980d-0501-49a4-bad0-4af78c316441","Type":"ContainerDied","Data":"727cfefd598a55982d0a06a798373ee45be90ae3b4652743344047549582caca"}
Oct 30 23:45:22.090013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.089977    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp"
Oct 30 23:45:22.094281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.094263    5078 scope.go:122] "RemoveContainer" containerID="9752c2bcb3915e92e338b31b835cf9772e479723795f8ec950a5c5fd73b56d87"
Oct 30 23:45:22.105226 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.105090    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs"
Oct 30 23:45:22.211938 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.211909    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl"
Oct 30 23:45:22.219002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.218976    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6416daa-de15-47d8-aca5-da1b417b6750-kube-api-access-trbs6\" (UniqueName: \"kubernetes.io/projected/b6416daa-de15-47d8-aca5-da1b417b6750-kube-api-access-trbs6\") pod \"b6416daa-de15-47d8-aca5-da1b417b6750\" (UID: \"b6416daa-de15-47d8-aca5-da1b417b6750\") "
Oct 30 23:45:22.219207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.219193    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6416daa-de15-47d8-aca5-da1b417b6750-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6416daa-de15-47d8-aca5-da1b417b6750-plugins-config-volume\") pod \"b6416daa-de15-47d8-aca5-da1b417b6750\" (UID: \"b6416daa-de15-47d8-aca5-da1b417b6750\") "
Oct 30 23:45:22.219555 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.219526    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6416daa-de15-47d8-aca5-da1b417b6750-plugins-config-volume" pod "b6416daa-de15-47d8-aca5-da1b417b6750" (UID: "b6416daa-de15-47d8-aca5-da1b417b6750"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:22.219687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.219671    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6416daa-de15-47d8-aca5-da1b417b6750-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:22.221076 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.221055    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6416daa-de15-47d8-aca5-da1b417b6750-kube-api-access-trbs6" pod "b6416daa-de15-47d8-aca5-da1b417b6750" (UID: "b6416daa-de15-47d8-aca5-da1b417b6750"). InnerVolumeSpecName "kube-api-access-trbs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:22.245386 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.245357    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp"]
Oct 30 23:45:22.251958 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.251932    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bcc619-29890065-xt9gp"]
Oct 30 23:45:22.298490 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.298458    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a24baa-07c5-4252-a4e9-8ca2dda0f829" path="/var/lib/kubelet/pods/00a24baa-07c5-4252-a4e9-8ca2dda0f829/volumes"
Oct 30 23:45:22.299040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.299019    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53dc275c-99f8-40a6-ae1e-7acb22c8718c" path="/var/lib/kubelet/pods/53dc275c-99f8-40a6-ae1e-7acb22c8718c/volumes"
Oct 30 23:45:22.299498 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.299479    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945e980d-0501-49a4-bad0-4af78c316441" path="/var/lib/kubelet/pods/945e980d-0501-49a4-bad0-4af78c316441/volumes"
Oct 30 23:45:22.300404 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.300381    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="977fb1cb-c631-476a-8a27-373ce7cace0a" path="/var/lib/kubelet/pods/977fb1cb-c631-476a-8a27-373ce7cace0a/volumes"
Oct 30 23:45:22.300928 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.300912    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acddcfa4-c6f5-413d-a172-74ec4ab2433f" path="/var/lib/kubelet/pods/acddcfa4-c6f5-413d-a172-74ec4ab2433f/volumes"
Oct 30 23:45:22.301291 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.301278    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca0cd089-df4b-4d76-854b-217b6c93b42c" path="/var/lib/kubelet/pods/ca0cd089-df4b-4d76-854b-217b6c93b42c/volumes"
Oct 30 23:45:22.336154 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.334021    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba44784b-5ea3-4e23-81a4-09e94531aceb-kube-api-access-tmcwc\" (UniqueName: \"kubernetes.io/projected/ba44784b-5ea3-4e23-81a4-09e94531aceb-kube-api-access-tmcwc\") pod \"ba44784b-5ea3-4e23-81a4-09e94531aceb\" (UID: \"ba44784b-5ea3-4e23-81a4-09e94531aceb\") "
Oct 30 23:45:22.336154 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.334164    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba44784b-5ea3-4e23-81a4-09e94531aceb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba44784b-5ea3-4e23-81a4-09e94531aceb-plugins-config-volume\") pod \"ba44784b-5ea3-4e23-81a4-09e94531aceb\" (UID: \"ba44784b-5ea3-4e23-81a4-09e94531aceb\") "
Oct 30 23:45:22.351634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.351485    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trbs6\" (UniqueName: \"kubernetes.io/projected/b6416daa-de15-47d8-aca5-da1b417b6750-kube-api-access-trbs6\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:22.352466 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.352212    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba44784b-5ea3-4e23-81a4-09e94531aceb-plugins-config-volume" pod "ba44784b-5ea3-4e23-81a4-09e94531aceb" (UID: "ba44784b-5ea3-4e23-81a4-09e94531aceb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:22.356188 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.354972    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba44784b-5ea3-4e23-81a4-09e94531aceb-kube-api-access-tmcwc" pod "ba44784b-5ea3-4e23-81a4-09e94531aceb" (UID: "ba44784b-5ea3-4e23-81a4-09e94531aceb"). InnerVolumeSpecName "kube-api-access-tmcwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:22.452269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.452239    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmcwc\" (UniqueName: \"kubernetes.io/projected/ba44784b-5ea3-4e23-81a4-09e94531aceb-kube-api-access-tmcwc\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:22.452269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.452266    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba44784b-5ea3-4e23-81a4-09e94531aceb-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:22.541621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.533333    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8"]
Oct 30 23:45:22.576053 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:22.562528    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-fb14dc-29890065-qlnz8"]
Oct 30 23:45:23.128144 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:23.128108    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs" event={"ID":"b6416daa-de15-47d8-aca5-da1b417b6750","Type":"ContainerDied","Data":"8765b870dd7539597641f3b89cb31eb1260c45a379821e06b781d2d6a3f4b20b"}
Oct 30 23:45:23.128144 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:23.128137    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs"
Oct 30 23:45:23.128708 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:23.128213    5078 scope.go:122] "RemoveContainer" containerID="1fefbc4127ab01f1682ca43b2837afd38f510d94b5135c29b32e843072e0851d"
Oct 30 23:45:23.132437 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:23.132402    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl"
Oct 30 23:45:23.132730 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:23.132424    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl" event={"ID":"ba44784b-5ea3-4e23-81a4-09e94531aceb","Type":"ContainerDied","Data":"c8e1de54cb5213ed048813b6133f995b6dbff9c84349bb60abfd02e7b6b34f1a"}
Oct 30 23:45:23.141498 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:23.141483    5078 scope.go:122] "RemoveContainer" containerID="308ab5f779e59d293c8474379980f14543ac24f520d857b3e9bc7d0326892d18"
Oct 30 23:45:23.295223 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:23.295188    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl"]
Oct 30 23:45:23.298975 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:23.298950    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-989a4f-56649cd95-v7xtl"]
Oct 30 23:45:23.414955 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:23.414879    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs"]
Oct 30 23:45:23.423254 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:23.423223    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-989a4f-df7fdb4bf-5d6xs"]
Oct 30 23:45:24.298941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:24.298894    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3349e30c-c66e-4f0f-b640-f0d1d0c50588" path="/var/lib/kubelet/pods/3349e30c-c66e-4f0f-b640-f0d1d0c50588/volumes"
Oct 30 23:45:24.299506 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:24.299484    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6416daa-de15-47d8-aca5-da1b417b6750" path="/var/lib/kubelet/pods/b6416daa-de15-47d8-aca5-da1b417b6750/volumes"
Oct 30 23:45:24.299943 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:24.299923    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba44784b-5ea3-4e23-81a4-09e94531aceb" path="/var/lib/kubelet/pods/ba44784b-5ea3-4e23-81a4-09e94531aceb/volumes"
Oct 30 23:45:26.104280 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:26.104249    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz"]
Oct 30 23:45:26.154004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:26.153975    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bcc619-g8gzz"]
Oct 30 23:45:26.297574 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:26.297542    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09f50622-c9d6-43c2-bde6-219e11207233" path="/var/lib/kubelet/pods/09f50622-c9d6-43c2-bde6-219e11207233/volumes"
Oct 30 23:45:30.298163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298133    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb"]
Oct 30 23:45:30.298567 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298555    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ca0cd089-df4b-4d76-854b-217b6c93b42c" containerName="cleanup"
Oct 30 23:45:30.298655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298569    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0cd089-df4b-4d76-854b-217b6c93b42c" containerName="cleanup"
Oct 30 23:45:30.298655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298581    5078 cpu_manager.go:414] "containerMap: removing container" podUID="945e980d-0501-49a4-bad0-4af78c316441" containerName="cleanup"
Oct 30 23:45:30.298655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298587    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945e980d-0501-49a4-bad0-4af78c316441" containerName="cleanup"
Oct 30 23:45:30.298655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298608    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3349e30c-c66e-4f0f-b640-f0d1d0c50588" containerName="cleanup"
Oct 30 23:45:30.298655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298613    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3349e30c-c66e-4f0f-b640-f0d1d0c50588" containerName="cleanup"
Oct 30 23:45:30.298655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298619    5078 cpu_manager.go:414] "containerMap: removing container" podUID="977fb1cb-c631-476a-8a27-373ce7cace0a" containerName="cleanup"
Oct 30 23:45:30.298655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298624    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="977fb1cb-c631-476a-8a27-373ce7cace0a" containerName="cleanup"
Oct 30 23:45:30.298655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298631    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ca0cd089-df4b-4d76-854b-217b6c93b42c" containerName="cleanup"
Oct 30 23:45:30.298655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298635    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0cd089-df4b-4d76-854b-217b6c93b42c" containerName="cleanup"
Oct 30 23:45:30.298655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298641    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7cfecc2c-1a44-4f1d-828f-69f08b03a3f4" containerName="revoke-keys"
Oct 30 23:45:30.298655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298645    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cfecc2c-1a44-4f1d-828f-69f08b03a3f4" containerName="revoke-keys"
Oct 30 23:45:30.298655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298654    5078 cpu_manager.go:414] "containerMap: removing container" podUID="00a24baa-07c5-4252-a4e9-8ca2dda0f829" containerName="cleanup"
Oct 30 23:45:30.298655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298658    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a24baa-07c5-4252-a4e9-8ca2dda0f829" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298666    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b96c7b9a-2055-4654-b26c-d370869a2f1d" containerName="revoke-keys"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298670    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b96c7b9a-2055-4654-b26c-d370869a2f1d" containerName="revoke-keys"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298676    5078 cpu_manager.go:414] "containerMap: removing container" podUID="00a24baa-07c5-4252-a4e9-8ca2dda0f829" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298680    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a24baa-07c5-4252-a4e9-8ca2dda0f829" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298686    5078 cpu_manager.go:414] "containerMap: removing container" podUID="00a24baa-07c5-4252-a4e9-8ca2dda0f829" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298690    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a24baa-07c5-4252-a4e9-8ca2dda0f829" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298696    5078 cpu_manager.go:414] "containerMap: removing container" podUID="acddcfa4-c6f5-413d-a172-74ec4ab2433f" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298700    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acddcfa4-c6f5-413d-a172-74ec4ab2433f" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298710    5078 cpu_manager.go:414] "containerMap: removing container" podUID="945e980d-0501-49a4-bad0-4af78c316441" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298714    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945e980d-0501-49a4-bad0-4af78c316441" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298723    5078 cpu_manager.go:414] "containerMap: removing container" podUID="53dc275c-99f8-40a6-ae1e-7acb22c8718c" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298727    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53dc275c-99f8-40a6-ae1e-7acb22c8718c" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298734    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b6416daa-de15-47d8-aca5-da1b417b6750" containerName="payload-processing"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298738    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6416daa-de15-47d8-aca5-da1b417b6750" containerName="payload-processing"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298744    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3349e30c-c66e-4f0f-b640-f0d1d0c50588" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298748    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3349e30c-c66e-4f0f-b640-f0d1d0c50588" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298758    5078 cpu_manager.go:414] "containerMap: removing container" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298763    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298769    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b6b6065b-de7d-41b2-9733-42f56d923d59" containerName="payload-pre-processing"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298773    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b6065b-de7d-41b2-9733-42f56d923d59" containerName="payload-pre-processing"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298783    5078 cpu_manager.go:414] "containerMap: removing container" podUID="6fca3a8d-ef11-4d95-ba93-bcafa869e2c1" containerName="payload-processing"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298787    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fca3a8d-ef11-4d95-ba93-bcafa869e2c1" containerName="payload-processing"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298794    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ba44784b-5ea3-4e23-81a4-09e94531aceb" containerName="payload-pre-processing"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298798    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba44784b-5ea3-4e23-81a4-09e94531aceb" containerName="payload-pre-processing"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298808    5078 cpu_manager.go:414] "containerMap: removing container" podUID="53dc275c-99f8-40a6-ae1e-7acb22c8718c" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298813    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53dc275c-99f8-40a6-ae1e-7acb22c8718c" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298824    5078 cpu_manager.go:414] "containerMap: removing container" podUID="53dc275c-99f8-40a6-ae1e-7acb22c8718c" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298828    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53dc275c-99f8-40a6-ae1e-7acb22c8718c" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298834    5078 cpu_manager.go:414] "containerMap: removing container" podUID="acddcfa4-c6f5-413d-a172-74ec4ab2433f" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298838    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acddcfa4-c6f5-413d-a172-74ec4ab2433f" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298846    5078 cpu_manager.go:414] "containerMap: removing container" podUID="977fb1cb-c631-476a-8a27-373ce7cace0a" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298850    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="977fb1cb-c631-476a-8a27-373ce7cace0a" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298915    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="acddcfa4-c6f5-413d-a172-74ec4ab2433f" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298923    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="acddcfa4-c6f5-413d-a172-74ec4ab2433f" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298931    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="977fb1cb-c631-476a-8a27-373ce7cace0a" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298937    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="945e980d-0501-49a4-bad0-4af78c316441" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298942    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0cd089-df4b-4d76-854b-217b6c93b42c" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298948    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="945e980d-0501-49a4-bad0-4af78c316441" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298953    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3349e30c-c66e-4f0f-b640-f0d1d0c50588" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298958    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="977fb1cb-c631-476a-8a27-373ce7cace0a" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298962    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0cd089-df4b-4d76-854b-217b6c93b42c" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298967    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cfecc2c-1a44-4f1d-828f-69f08b03a3f4" containerName="revoke-keys"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298973    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a24baa-07c5-4252-a4e9-8ca2dda0f829" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298978    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b96c7b9a-2055-4654-b26c-d370869a2f1d" containerName="revoke-keys"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298982    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca0cd089-df4b-4d76-854b-217b6c93b42c" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298987    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a24baa-07c5-4252-a4e9-8ca2dda0f829" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298993    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a24baa-07c5-4252-a4e9-8ca2dda0f829" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.298998    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="945e980d-0501-49a4-bad0-4af78c316441" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.299003    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="53dc275c-99f8-40a6-ae1e-7acb22c8718c" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.299007    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6416daa-de15-47d8-aca5-da1b417b6750" containerName="payload-processing"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.299012    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4aa2ef-68c0-4d5d-b966-6c2f65766f7d" containerName="main"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.299017    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b6065b-de7d-41b2-9733-42f56d923d59" containerName="payload-pre-processing"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.299023    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fca3a8d-ef11-4d95-ba93-bcafa869e2c1" containerName="payload-processing"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.299029    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba44784b-5ea3-4e23-81a4-09e94531aceb" containerName="payload-pre-processing"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.299033    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="53dc275c-99f8-40a6-ae1e-7acb22c8718c" containerName="cleanup"
Oct 30 23:45:30.299055 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.299038    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3349e30c-c66e-4f0f-b640-f0d1d0c50588" containerName="cleanup"
Oct 30 23:45:30.303986 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.303971    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb"
Oct 30 23:45:30.339801 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.339772    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8"]
Oct 30 23:45:30.340222 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.340210    5078 cpu_manager.go:414] "containerMap: removing container" podUID="977fb1cb-c631-476a-8a27-373ce7cace0a" containerName="cleanup"
Oct 30 23:45:30.340266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.340224    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="977fb1cb-c631-476a-8a27-373ce7cace0a" containerName="cleanup"
Oct 30 23:45:30.340266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.340234    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ca0cd089-df4b-4d76-854b-217b6c93b42c" containerName="cleanup"
Oct 30 23:45:30.340266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.340239    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca0cd089-df4b-4d76-854b-217b6c93b42c" containerName="cleanup"
Oct 30 23:45:30.340266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.340248    5078 cpu_manager.go:414] "containerMap: removing container" podUID="acddcfa4-c6f5-413d-a172-74ec4ab2433f" containerName="cleanup"
Oct 30 23:45:30.340266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.340254    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acddcfa4-c6f5-413d-a172-74ec4ab2433f" containerName="cleanup"
Oct 30 23:45:30.340453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.340267    5078 cpu_manager.go:414] "containerMap: removing container" podUID="945e980d-0501-49a4-bad0-4af78c316441" containerName="cleanup"
Oct 30 23:45:30.340453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.340272    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945e980d-0501-49a4-bad0-4af78c316441" containerName="cleanup"
Oct 30 23:45:30.340453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.340284    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3349e30c-c66e-4f0f-b640-f0d1d0c50588" containerName="cleanup"
Oct 30 23:45:30.340453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.340288    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3349e30c-c66e-4f0f-b640-f0d1d0c50588" containerName="cleanup"
Oct 30 23:45:30.340453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.340373    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="53dc275c-99f8-40a6-ae1e-7acb22c8718c" containerName="cleanup"
Oct 30 23:45:30.340453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.340382    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3349e30c-c66e-4f0f-b640-f0d1d0c50588" containerName="cleanup"
Oct 30 23:45:30.340453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.340390    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="acddcfa4-c6f5-413d-a172-74ec4ab2433f" containerName="cleanup"
Oct 30 23:45:30.340453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.340398    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="977fb1cb-c631-476a-8a27-373ce7cace0a" containerName="cleanup"
Oct 30 23:45:30.343779 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.343761    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8"
Oct 30 23:45:30.353131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.353108    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb"]
Oct 30 23:45:30.386893 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.386860    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:45:30.387148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.386890    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:45:30.387148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.386917    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:45:30.393210 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.393179    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f7a9791-44mlb\" (UID: \"a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb"
Oct 30 23:45:30.393396 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.393379    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f7a9791-44mlb\" (UID: \"a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb"
Oct 30 23:45:30.393474 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.393433    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7dfj\" (UniqueName: \"kubernetes.io/projected/3a07d752-751e-4a29-8a8d-f23d2144b3ba-kube-api-access-j7dfj\") pod \"maas-api-revoke-sub-984b7668-d89h8\" (UID: \"3a07d752-751e-4a29-8a8d-f23d2144b3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8"
Oct 30 23:45:30.393758 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.393722    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zvzv\" (UniqueName: \"kubernetes.io/projected/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-kube-api-access-9zvzv\") pod \"maas-api-revoke-sub-1f7a9791-44mlb\" (UID: \"a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb"
Oct 30 23:45:30.393861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.393777    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a07d752-751e-4a29-8a8d-f23d2144b3ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-984b7668-d89h8\" (UID: \"3a07d752-751e-4a29-8a8d-f23d2144b3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8"
Oct 30 23:45:30.393861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.393834    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a07d752-751e-4a29-8a8d-f23d2144b3ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-984b7668-d89h8\" (UID: \"3a07d752-751e-4a29-8a8d-f23d2144b3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8"
Oct 30 23:45:30.398661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.398641    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:45:30.403318 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.403051    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8"]
Oct 30 23:45:30.495105 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.495068    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f7a9791-44mlb\" (UID: \"a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb"
Oct 30 23:45:30.495292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.495138    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f7a9791-44mlb\" (UID: \"a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb"
Oct 30 23:45:30.495292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.495162    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7dfj\" (UniqueName: \"kubernetes.io/projected/3a07d752-751e-4a29-8a8d-f23d2144b3ba-kube-api-access-j7dfj\") pod \"maas-api-revoke-sub-984b7668-d89h8\" (UID: \"3a07d752-751e-4a29-8a8d-f23d2144b3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8"
Oct 30 23:45:30.495292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.495213    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zvzv\" (UniqueName: \"kubernetes.io/projected/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-kube-api-access-9zvzv\") pod \"maas-api-revoke-sub-1f7a9791-44mlb\" (UID: \"a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb"
Oct 30 23:45:30.495460 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.495337    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a07d752-751e-4a29-8a8d-f23d2144b3ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-984b7668-d89h8\" (UID: \"3a07d752-751e-4a29-8a8d-f23d2144b3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8"
Oct 30 23:45:30.495460 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.495378    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a07d752-751e-4a29-8a8d-f23d2144b3ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-984b7668-d89h8\" (UID: \"3a07d752-751e-4a29-8a8d-f23d2144b3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8"
Oct 30 23:45:30.495933 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.495904    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f7a9791-44mlb\" (UID: \"a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb"
Oct 30 23:45:30.496068 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.496036    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a07d752-751e-4a29-8a8d-f23d2144b3ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-984b7668-d89h8\" (UID: \"3a07d752-751e-4a29-8a8d-f23d2144b3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8"
Oct 30 23:45:30.497713 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.497695    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f7a9791-44mlb\" (UID: \"a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb"
Oct 30 23:45:30.497934 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.497918    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a07d752-751e-4a29-8a8d-f23d2144b3ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-984b7668-d89h8\" (UID: \"3a07d752-751e-4a29-8a8d-f23d2144b3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8"
Oct 30 23:45:30.551945 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.551876    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zvzv\" (UniqueName: \"kubernetes.io/projected/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-kube-api-access-9zvzv\") pod \"maas-api-revoke-sub-1f7a9791-44mlb\" (UID: \"a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb"
Oct 30 23:45:30.552092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.552073    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7dfj\" (UniqueName: \"kubernetes.io/projected/3a07d752-751e-4a29-8a8d-f23d2144b3ba-kube-api-access-j7dfj\") pod \"maas-api-revoke-sub-984b7668-d89h8\" (UID: \"3a07d752-751e-4a29-8a8d-f23d2144b3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8"
Oct 30 23:45:30.617156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.617123    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb"
Oct 30 23:45:30.658768 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:30.658735    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8"
Oct 30 23:45:30.999651 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:45:30.999612    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ab1f5f_0fab_42cc_8e82_178a0cbcc07b.slice/crio-af375da3d42872aeaa4c7fe83e13b2d7a93ed3dd4103f035691d53893d83e7a0 WatchSource:0}: Error finding container af375da3d42872aeaa4c7fe83e13b2d7a93ed3dd4103f035691d53893d83e7a0: Status 404 returned error can't find the container with id af375da3d42872aeaa4c7fe83e13b2d7a93ed3dd4103f035691d53893d83e7a0
Oct 30 23:45:31.000909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:31.000878    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb"]
Oct 30 23:45:31.057111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:31.057084    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8"]
Oct 30 23:45:31.057975 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:45:31.057943    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a07d752_751e_4a29_8a8d_f23d2144b3ba.slice/crio-71923de920d7a9d14b5f3c11e9e89f871fd8c22a91ff0d319f06852bdce8701d WatchSource:0}: Error finding container 71923de920d7a9d14b5f3c11e9e89f871fd8c22a91ff0d319f06852bdce8701d: Status 404 returned error can't find the container with id 71923de920d7a9d14b5f3c11e9e89f871fd8c22a91ff0d319f06852bdce8701d
Oct 30 23:45:31.192682 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:31.192645    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8" event={"ID":"3a07d752-751e-4a29-8a8d-f23d2144b3ba","Type":"ContainerStarted","Data":"71923de920d7a9d14b5f3c11e9e89f871fd8c22a91ff0d319f06852bdce8701d"}
Oct 30 23:45:31.193907 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:31.193884    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb" event={"ID":"a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b","Type":"ContainerStarted","Data":"af375da3d42872aeaa4c7fe83e13b2d7a93ed3dd4103f035691d53893d83e7a0"}
Oct 30 23:45:32.201838 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:32.201764    5078 generic.go:356] "Generic (PLEG): container finished" podID="3a07d752-751e-4a29-8a8d-f23d2144b3ba" containerID="03c55032ec0b0e56ae8ff5591591972df192d30dfb122bd6215f26d77647329b" exitCode=0
Oct 30 23:45:32.202226 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:32.201852    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8" event={"ID":"3a07d752-751e-4a29-8a8d-f23d2144b3ba","Type":"ContainerDied","Data":"03c55032ec0b0e56ae8ff5591591972df192d30dfb122bd6215f26d77647329b"}
Oct 30 23:45:32.203401 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:32.203382    5078 generic.go:356] "Generic (PLEG): container finished" podID="a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b" containerID="22f9f7d86b11123e33bb9335b111165225a4bcad20d35e7455f14ac121aa546d" exitCode=0
Oct 30 23:45:32.203502 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:32.203442    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb" event={"ID":"a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b","Type":"ContainerDied","Data":"22f9f7d86b11123e33bb9335b111165225a4bcad20d35e7455f14ac121aa546d"}
Oct 30 23:45:32.923252 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:32.923221    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"]
Oct 30 23:45:32.923455 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:32.923434    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4" podUID="eb7f4356-b694-4ebb-a65e-8f7782591aea" containerName="istio-proxy" containerID="cri-o://0892ed9f3ad748423685174496cd366240b726ecf65c98508490c3d22111b935" gracePeriod=30
Oct 30 23:45:33.385936 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.385910    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8"
Oct 30 23:45:33.427252 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.427227    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb"
Oct 30 23:45:33.431619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.431570    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a07d752-751e-4a29-8a8d-f23d2144b3ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a07d752-751e-4a29-8a8d-f23d2144b3ba-maas-api-xks-ca\") pod \"3a07d752-751e-4a29-8a8d-f23d2144b3ba\" (UID: \"3a07d752-751e-4a29-8a8d-f23d2144b3ba\") "
Oct 30 23:45:33.431710 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.431632    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a07d752-751e-4a29-8a8d-f23d2144b3ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a07d752-751e-4a29-8a8d-f23d2144b3ba-maas-api-openshift-ca\") pod \"3a07d752-751e-4a29-8a8d-f23d2144b3ba\" (UID: \"3a07d752-751e-4a29-8a8d-f23d2144b3ba\") "
Oct 30 23:45:33.431759 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.431733    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a07d752-751e-4a29-8a8d-f23d2144b3ba-kube-api-access-j7dfj\" (UniqueName: \"kubernetes.io/projected/3a07d752-751e-4a29-8a8d-f23d2144b3ba-kube-api-access-j7dfj\") pod \"3a07d752-751e-4a29-8a8d-f23d2144b3ba\" (UID: \"3a07d752-751e-4a29-8a8d-f23d2144b3ba\") "
Oct 30 23:45:33.432007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.431987    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a07d752-751e-4a29-8a8d-f23d2144b3ba-maas-api-openshift-ca" pod "3a07d752-751e-4a29-8a8d-f23d2144b3ba" (UID: "3a07d752-751e-4a29-8a8d-f23d2144b3ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:33.432246 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.432200    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a07d752-751e-4a29-8a8d-f23d2144b3ba-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:33.433790 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.433768    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a07d752-751e-4a29-8a8d-f23d2144b3ba-maas-api-xks-ca" pod "3a07d752-751e-4a29-8a8d-f23d2144b3ba" (UID: "3a07d752-751e-4a29-8a8d-f23d2144b3ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:45:33.433960 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.433944    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a07d752-751e-4a29-8a8d-f23d2144b3ba-kube-api-access-j7dfj" pod "3a07d752-751e-4a29-8a8d-f23d2144b3ba" (UID: "3a07d752-751e-4a29-8a8d-f23d2144b3ba"). InnerVolumeSpecName "kube-api-access-j7dfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:33.532849 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.532761    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-maas-api-openshift-ca\") pod \"a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b\" (UID: \"a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b\") "
Oct 30 23:45:33.533022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.532949    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-kube-api-access-9zvzv\" (UniqueName: \"kubernetes.io/projected/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-kube-api-access-9zvzv\") pod \"a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b\" (UID: \"a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b\") "
Oct 30 23:45:33.533093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.533069    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-maas-api-xks-ca\") pod \"a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b\" (UID: \"a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b\") "
Oct 30 23:45:33.533211 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.533181    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-maas-api-openshift-ca" pod "a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b" (UID: "a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:33.533533 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.533511    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7dfj\" (UniqueName: \"kubernetes.io/projected/3a07d752-751e-4a29-8a8d-f23d2144b3ba-kube-api-access-j7dfj\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:33.533855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.533540    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:33.533855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.533555    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a07d752-751e-4a29-8a8d-f23d2144b3ba-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:33.535269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.535244    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-kube-api-access-9zvzv" pod "a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b" (UID: "a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b"). InnerVolumeSpecName "kube-api-access-9zvzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:33.535926 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.535903    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-maas-api-xks-ca" pod "a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b" (UID: "a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:45:33.634645 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.634590    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:33.634807 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.634648    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zvzv\" (UniqueName: \"kubernetes.io/projected/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b-kube-api-access-9zvzv\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:33.735422 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.735390    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89"]
Oct 30 23:45:33.736023 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.735961    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b" containerName="revoke-keys"
Oct 30 23:45:33.736126 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.736039    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b" containerName="revoke-keys"
Oct 30 23:45:33.736126 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.736073    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3a07d752-751e-4a29-8a8d-f23d2144b3ba" containerName="revoke-keys"
Oct 30 23:45:33.736126 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.736078    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a07d752-751e-4a29-8a8d-f23d2144b3ba" containerName="revoke-keys"
Oct 30 23:45:33.736263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.736182    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b" containerName="revoke-keys"
Oct 30 23:45:33.736263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.736195    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a07d752-751e-4a29-8a8d-f23d2144b3ba" containerName="revoke-keys"
Oct 30 23:45:33.739705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.739683    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89"
Oct 30 23:45:33.795571 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.795539    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89"]
Oct 30 23:45:33.836568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.836538    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj5lx\" (UniqueName: \"kubernetes.io/projected/d365bd08-d827-41b2-83ba-c244d745c64e-kube-api-access-lj5lx\") pod \"maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89\" (UID: \"d365bd08-d827-41b2-83ba-c244d745c64e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89"
Oct 30 23:45:33.836728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.836589    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d365bd08-d827-41b2-83ba-c244d745c64e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89\" (UID: \"d365bd08-d827-41b2-83ba-c244d745c64e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89"
Oct 30 23:45:33.836728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.836673    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d365bd08-d827-41b2-83ba-c244d745c64e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89\" (UID: \"d365bd08-d827-41b2-83ba-c244d745c64e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89"
Oct 30 23:45:33.937976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.937946    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj5lx\" (UniqueName: \"kubernetes.io/projected/d365bd08-d827-41b2-83ba-c244d745c64e-kube-api-access-lj5lx\") pod \"maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89\" (UID: \"d365bd08-d827-41b2-83ba-c244d745c64e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89"
Oct 30 23:45:33.938177 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.937998    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d365bd08-d827-41b2-83ba-c244d745c64e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89\" (UID: \"d365bd08-d827-41b2-83ba-c244d745c64e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89"
Oct 30 23:45:33.938177 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.938020    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d365bd08-d827-41b2-83ba-c244d745c64e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89\" (UID: \"d365bd08-d827-41b2-83ba-c244d745c64e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89"
Oct 30 23:45:33.938719 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.938694    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d365bd08-d827-41b2-83ba-c244d745c64e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89\" (UID: \"d365bd08-d827-41b2-83ba-c244d745c64e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89"
Oct 30 23:45:33.940507 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.940487    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d365bd08-d827-41b2-83ba-c244d745c64e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89\" (UID: \"d365bd08-d827-41b2-83ba-c244d745c64e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89"
Oct 30 23:45:33.994295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:33.994263    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj5lx\" (UniqueName: \"kubernetes.io/projected/d365bd08-d827-41b2-83ba-c244d745c64e-kube-api-access-lj5lx\") pod \"maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89\" (UID: \"d365bd08-d827-41b2-83ba-c244d745c64e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89"
Oct 30 23:45:34.053290 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:34.053217    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89"
Oct 30 23:45:34.217555 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:34.217527    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8"
Oct 30 23:45:34.217555 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:34.217533    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8" event={"ID":"3a07d752-751e-4a29-8a8d-f23d2144b3ba","Type":"ContainerDied","Data":"71923de920d7a9d14b5f3c11e9e89f871fd8c22a91ff0d319f06852bdce8701d"}
Oct 30 23:45:34.217773 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:34.217583    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71923de920d7a9d14b5f3c11e9e89f871fd8c22a91ff0d319f06852bdce8701d"
Oct 30 23:45:34.219029 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:34.219007    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb"
Oct 30 23:45:34.219207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:34.219059    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb" event={"ID":"a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b","Type":"ContainerDied","Data":"af375da3d42872aeaa4c7fe83e13b2d7a93ed3dd4103f035691d53893d83e7a0"}
Oct 30 23:45:34.219207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:34.219106    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af375da3d42872aeaa4c7fe83e13b2d7a93ed3dd4103f035691d53893d83e7a0"
Oct 30 23:45:34.474509 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:34.474488    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89"]
Oct 30 23:45:34.476214 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:45:34.476183    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd365bd08_d827_41b2_83ba_c244d745c64e.slice/crio-683b09331bb087766a47ef8debeb47bf2a4b3dcb6213237803cd0f1e4bdf7a85 WatchSource:0}: Error finding container 683b09331bb087766a47ef8debeb47bf2a4b3dcb6213237803cd0f1e4bdf7a85: Status 404 returned error can't find the container with id 683b09331bb087766a47ef8debeb47bf2a4b3dcb6213237803cd0f1e4bdf7a85
Oct 30 23:45:35.231795 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:35.231766    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89" event={"ID":"d365bd08-d827-41b2-83ba-c244d745c64e","Type":"ContainerStarted","Data":"832b10462e75a568538368160dfd4cc397036591b914377f717617e1cdf044e3"}
Oct 30 23:45:35.231971 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:35.231829    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89" event={"ID":"d365bd08-d827-41b2-83ba-c244d745c64e","Type":"ContainerStarted","Data":"683b09331bb087766a47ef8debeb47bf2a4b3dcb6213237803cd0f1e4bdf7a85"}
Oct 30 23:45:35.347629 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:35.347560    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89" podStartSLOduration=2.347542921 podStartE2EDuration="2.347542921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:45:35.34476034 +0000 UTC m=+2321.232666693" watchObservedRunningTime="2026-10-30 23:45:35.347542921 +0000 UTC m=+2321.235449268"
Oct 30 23:45:36.247294 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:36.247262    5078 generic.go:356] "Generic (PLEG): container finished" podID="d365bd08-d827-41b2-83ba-c244d745c64e" containerID="832b10462e75a568538368160dfd4cc397036591b914377f717617e1cdf044e3" exitCode=0
Oct 30 23:45:36.247701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:36.247349    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89" event={"ID":"d365bd08-d827-41b2-83ba-c244d745c64e","Type":"ContainerDied","Data":"832b10462e75a568538368160dfd4cc397036591b914377f717617e1cdf044e3"}
Oct 30 23:45:37.396489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:37.396466    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89"
Oct 30 23:45:37.527267 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:37.527192    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d365bd08-d827-41b2-83ba-c244d745c64e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d365bd08-d827-41b2-83ba-c244d745c64e-maas-api-xks-ca\") pod \"d365bd08-d827-41b2-83ba-c244d745c64e\" (UID: \"d365bd08-d827-41b2-83ba-c244d745c64e\") "
Oct 30 23:45:37.527411 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:37.527294    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d365bd08-d827-41b2-83ba-c244d745c64e-kube-api-access-lj5lx\" (UniqueName: \"kubernetes.io/projected/d365bd08-d827-41b2-83ba-c244d745c64e-kube-api-access-lj5lx\") pod \"d365bd08-d827-41b2-83ba-c244d745c64e\" (UID: \"d365bd08-d827-41b2-83ba-c244d745c64e\") "
Oct 30 23:45:37.527454 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:37.527408    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d365bd08-d827-41b2-83ba-c244d745c64e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d365bd08-d827-41b2-83ba-c244d745c64e-maas-api-openshift-ca\") pod \"d365bd08-d827-41b2-83ba-c244d745c64e\" (UID: \"d365bd08-d827-41b2-83ba-c244d745c64e\") "
Oct 30 23:45:37.527827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:37.527795    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d365bd08-d827-41b2-83ba-c244d745c64e-maas-api-openshift-ca" pod "d365bd08-d827-41b2-83ba-c244d745c64e" (UID: "d365bd08-d827-41b2-83ba-c244d745c64e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:37.528274 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:37.528249    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d365bd08-d827-41b2-83ba-c244d745c64e-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:37.529646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:37.529570    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d365bd08-d827-41b2-83ba-c244d745c64e-kube-api-access-lj5lx" pod "d365bd08-d827-41b2-83ba-c244d745c64e" (UID: "d365bd08-d827-41b2-83ba-c244d745c64e"). InnerVolumeSpecName "kube-api-access-lj5lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:37.529921 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:37.529903    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d365bd08-d827-41b2-83ba-c244d745c64e-maas-api-xks-ca" pod "d365bd08-d827-41b2-83ba-c244d745c64e" (UID: "d365bd08-d827-41b2-83ba-c244d745c64e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:45:37.629207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:37.629169    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d365bd08-d827-41b2-83ba-c244d745c64e-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:37.629207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:37.629200    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj5lx\" (UniqueName: \"kubernetes.io/projected/d365bd08-d827-41b2-83ba-c244d745c64e-kube-api-access-lj5lx\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:38.172575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.172552    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:45:38.261719 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.261626    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89" event={"ID":"d365bd08-d827-41b2-83ba-c244d745c64e","Type":"ContainerDied","Data":"683b09331bb087766a47ef8debeb47bf2a4b3dcb6213237803cd0f1e4bdf7a85"}
Oct 30 23:45:38.261719 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.261691    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="683b09331bb087766a47ef8debeb47bf2a4b3dcb6213237803cd0f1e4bdf7a85"
Oct 30 23:45:38.261719 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.261639    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89"
Oct 30 23:45:38.263478 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.263455    5078 generic.go:356] "Generic (PLEG): container finished" podID="eb7f4356-b694-4ebb-a65e-8f7782591aea" containerID="0892ed9f3ad748423685174496cd366240b726ecf65c98508490c3d22111b935" exitCode=0
Oct 30 23:45:38.263664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.263493    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4" event={"ID":"eb7f4356-b694-4ebb-a65e-8f7782591aea","Type":"ContainerDied","Data":"0892ed9f3ad748423685174496cd366240b726ecf65c98508490c3d22111b935"}
Oct 30 23:45:38.263664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.263504    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"
Oct 30 23:45:38.263664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.263547    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4" event={"ID":"eb7f4356-b694-4ebb-a65e-8f7782591aea","Type":"ContainerDied","Data":"66486bbcd57086f4b6a7c5fce7c2b52c480103a35e0425b0163f53103ee5a4e1"}
Oct 30 23:45:38.263664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.263577    5078 scope.go:122] "RemoveContainer" containerID="0892ed9f3ad748423685174496cd366240b726ecf65c98508490c3d22111b935"
Oct 30 23:45:38.273726 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.273705    5078 scope.go:122] "RemoveContainer" containerID="f430980ddda5807584ae8cf577187e1ee40beb081b49c4c82c969495242684b9"
Oct 30 23:45:38.283426 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.283403    5078 scope.go:122] "RemoveContainer" containerID="0892ed9f3ad748423685174496cd366240b726ecf65c98508490c3d22111b935"
Oct 30 23:45:38.283750 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:38.283728    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0892ed9f3ad748423685174496cd366240b726ecf65c98508490c3d22111b935\": container with ID starting with 0892ed9f3ad748423685174496cd366240b726ecf65c98508490c3d22111b935 not found: ID does not exist" containerID="0892ed9f3ad748423685174496cd366240b726ecf65c98508490c3d22111b935"
Oct 30 23:45:38.283856 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.283761    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0892ed9f3ad748423685174496cd366240b726ecf65c98508490c3d22111b935"} err="failed to get container status \"0892ed9f3ad748423685174496cd366240b726ecf65c98508490c3d22111b935\": rpc error: code = NotFound desc = could not find container \"0892ed9f3ad748423685174496cd366240b726ecf65c98508490c3d22111b935\": container with ID starting with 0892ed9f3ad748423685174496cd366240b726ecf65c98508490c3d22111b935 not found: ID does not exist"
Oct 30 23:45:38.283856 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.283779    5078 scope.go:122] "RemoveContainer" containerID="f430980ddda5807584ae8cf577187e1ee40beb081b49c4c82c969495242684b9"
Oct 30 23:45:38.284089 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:38.284071    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f430980ddda5807584ae8cf577187e1ee40beb081b49c4c82c969495242684b9\": container with ID starting with f430980ddda5807584ae8cf577187e1ee40beb081b49c4c82c969495242684b9 not found: ID does not exist" containerID="f430980ddda5807584ae8cf577187e1ee40beb081b49c4c82c969495242684b9"
Oct 30 23:45:38.284133 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.284095    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f430980ddda5807584ae8cf577187e1ee40beb081b49c4c82c969495242684b9"} err="failed to get container status \"f430980ddda5807584ae8cf577187e1ee40beb081b49c4c82c969495242684b9\": rpc error: code = NotFound desc = could not find container \"f430980ddda5807584ae8cf577187e1ee40beb081b49c4c82c969495242684b9\": container with ID starting with f430980ddda5807584ae8cf577187e1ee40beb081b49c4c82c969495242684b9 not found: ID does not exist"
Oct 30 23:45:38.338003 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.337977    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-data\") pod \"eb7f4356-b694-4ebb-a65e-8f7782591aea\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") "
Oct 30 23:45:38.338166 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.338058    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-workload-certs\") pod \"eb7f4356-b694-4ebb-a65e-8f7782591aea\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") "
Oct 30 23:45:38.338166 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.338156    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-envoy\") pod \"eb7f4356-b694-4ebb-a65e-8f7782591aea\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") "
Oct 30 23:45:38.338269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.338226    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-token\" (UniqueName: \"kubernetes.io/projected/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-token\") pod \"eb7f4356-b694-4ebb-a65e-8f7782591aea\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") "
Oct 30 23:45:38.338492 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.338259    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-data" pod "eb7f4356-b694-4ebb-a65e-8f7782591aea" (UID: "eb7f4356-b694-4ebb-a65e-8f7782591aea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:45:38.338492 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.338295    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb7f4356-b694-4ebb-a65e-8f7782591aea-kube-api-access-58cdz\" (UniqueName: \"kubernetes.io/projected/eb7f4356-b694-4ebb-a65e-8f7782591aea-kube-api-access-58cdz\") pod \"eb7f4356-b694-4ebb-a65e-8f7782591aea\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") "
Oct 30 23:45:38.338492 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.338453    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb7f4356-b694-4ebb-a65e-8f7782591aea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7f4356-b694-4ebb-a65e-8f7782591aea-istiod-ca-cert\") pod \"eb7f4356-b694-4ebb-a65e-8f7782591aea\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") "
Oct 30 23:45:38.338492 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.338462    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-workload-certs" pod "eb7f4356-b694-4ebb-a65e-8f7782591aea" (UID: "eb7f4356-b694-4ebb-a65e-8f7782591aea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:45:38.338893 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.338505    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-credential-socket\") pod \"eb7f4356-b694-4ebb-a65e-8f7782591aea\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") "
Oct 30 23:45:38.338893 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.338543    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-workload-socket\") pod \"eb7f4356-b694-4ebb-a65e-8f7782591aea\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") "
Oct 30 23:45:38.338893 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.338689    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-podinfo\") pod \"eb7f4356-b694-4ebb-a65e-8f7782591aea\" (UID: \"eb7f4356-b694-4ebb-a65e-8f7782591aea\") "
Oct 30 23:45:38.339111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.338944    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb7f4356-b694-4ebb-a65e-8f7782591aea-istiod-ca-cert" pod "eb7f4356-b694-4ebb-a65e-8f7782591aea" (UID: "eb7f4356-b694-4ebb-a65e-8f7782591aea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:38.339234 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.339114    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-credential-socket" pod "eb7f4356-b694-4ebb-a65e-8f7782591aea" (UID: "eb7f4356-b694-4ebb-a65e-8f7782591aea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:45:38.339234 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.339198    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-workload-socket" pod "eb7f4356-b694-4ebb-a65e-8f7782591aea" (UID: "eb7f4356-b694-4ebb-a65e-8f7782591aea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:45:38.339625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.339578    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eb7f4356-b694-4ebb-a65e-8f7782591aea-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:38.339625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.339622    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:38.339862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.339651    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:38.339862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.339669    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:38.339862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.339708    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:38.341087 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.341057    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7f4356-b694-4ebb-a65e-8f7782591aea-kube-api-access-58cdz" pod "eb7f4356-b694-4ebb-a65e-8f7782591aea" (UID: "eb7f4356-b694-4ebb-a65e-8f7782591aea"). InnerVolumeSpecName "kube-api-access-58cdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:38.341087 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.341076    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-envoy" pod "eb7f4356-b694-4ebb-a65e-8f7782591aea" (UID: "eb7f4356-b694-4ebb-a65e-8f7782591aea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:45:38.341280 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.341263    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-token" pod "eb7f4356-b694-4ebb-a65e-8f7782591aea" (UID: "eb7f4356-b694-4ebb-a65e-8f7782591aea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:38.341340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.341322    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-podinfo" pod "eb7f4356-b694-4ebb-a65e-8f7782591aea" (UID: "eb7f4356-b694-4ebb-a65e-8f7782591aea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:45:38.441134 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.441104    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:38.441134 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.441132    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:38.441134 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.441141    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eb7f4356-b694-4ebb-a65e-8f7782591aea-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:38.441553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.441151    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58cdz\" (UniqueName: \"kubernetes.io/projected/eb7f4356-b694-4ebb-a65e-8f7782591aea-kube-api-access-58cdz\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:38.684994 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.684936    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"]
Oct 30 23:45:38.688367 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:38.688342    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-bcc619-openshift-default-5578794cc7-d4zb4"]
Oct 30 23:45:40.299102 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:40.299072    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb7f4356-b694-4ebb-a65e-8f7782591aea" path="/var/lib/kubelet/pods/eb7f4356-b694-4ebb-a65e-8f7782591aea/volumes"
Oct 30 23:45:45.278622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:45.273655    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j"]
Oct 30 23:45:47.592564 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:47.592517    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:45:50.274988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:50.274950    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j" podUID="33314a36-830a-4395-8f74-89227b549913" containerName="payload-processing" containerID="cri-o://ab7978292402368340713e4f0d991d4cabee65741fd57e5a0388d70abf3b246d" gracePeriod=25
Oct 30 23:45:50.539647 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:50.539558    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j"
Oct 30 23:45:50.611312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:50.611281    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33314a36-830a-4395-8f74-89227b549913-kube-api-access-4kht5\" (UniqueName: \"kubernetes.io/projected/33314a36-830a-4395-8f74-89227b549913-kube-api-access-4kht5\") pod \"33314a36-830a-4395-8f74-89227b549913\" (UID: \"33314a36-830a-4395-8f74-89227b549913\") "
Oct 30 23:45:50.611516 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:50.611436    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33314a36-830a-4395-8f74-89227b549913-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33314a36-830a-4395-8f74-89227b549913-plugins-config-volume\") pod \"33314a36-830a-4395-8f74-89227b549913\" (UID: \"33314a36-830a-4395-8f74-89227b549913\") "
Oct 30 23:45:50.611806 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:50.611784    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33314a36-830a-4395-8f74-89227b549913-plugins-config-volume" pod "33314a36-830a-4395-8f74-89227b549913" (UID: "33314a36-830a-4395-8f74-89227b549913"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:45:50.613542 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:50.613521    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33314a36-830a-4395-8f74-89227b549913-kube-api-access-4kht5" pod "33314a36-830a-4395-8f74-89227b549913" (UID: "33314a36-830a-4395-8f74-89227b549913"). InnerVolumeSpecName "kube-api-access-4kht5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:45:50.713169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:50.713137    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33314a36-830a-4395-8f74-89227b549913-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:50.713169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:50.713165    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kht5\" (UniqueName: \"kubernetes.io/projected/33314a36-830a-4395-8f74-89227b549913-kube-api-access-4kht5\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:45:51.382169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:51.382132    5078 generic.go:356] "Generic (PLEG): container finished" podID="33314a36-830a-4395-8f74-89227b549913" containerID="ab7978292402368340713e4f0d991d4cabee65741fd57e5a0388d70abf3b246d" exitCode=0
Oct 30 23:45:51.382665 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:51.382192    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j"
Oct 30 23:45:51.382665 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:51.382213    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j" event={"ID":"33314a36-830a-4395-8f74-89227b549913","Type":"ContainerDied","Data":"ab7978292402368340713e4f0d991d4cabee65741fd57e5a0388d70abf3b246d"}
Oct 30 23:45:51.382665 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:51.382269    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j" event={"ID":"33314a36-830a-4395-8f74-89227b549913","Type":"ContainerDied","Data":"0729890d198745fd220d8f9c17a458d93e687e84b03f84606b7dbdd2c20f77e4"}
Oct 30 23:45:51.382665 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:51.382293    5078 scope.go:122] "RemoveContainer" containerID="ab7978292402368340713e4f0d991d4cabee65741fd57e5a0388d70abf3b246d"
Oct 30 23:45:51.392770 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:51.392747    5078 scope.go:122] "RemoveContainer" containerID="ab7978292402368340713e4f0d991d4cabee65741fd57e5a0388d70abf3b246d"
Oct 30 23:45:51.393072 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:45:51.393043    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab7978292402368340713e4f0d991d4cabee65741fd57e5a0388d70abf3b246d\": container with ID starting with ab7978292402368340713e4f0d991d4cabee65741fd57e5a0388d70abf3b246d not found: ID does not exist" containerID="ab7978292402368340713e4f0d991d4cabee65741fd57e5a0388d70abf3b246d"
Oct 30 23:45:51.393124 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:51.393084    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab7978292402368340713e4f0d991d4cabee65741fd57e5a0388d70abf3b246d"} err="failed to get container status \"ab7978292402368340713e4f0d991d4cabee65741fd57e5a0388d70abf3b246d\": rpc error: code = NotFound desc = could not find container \"ab7978292402368340713e4f0d991d4cabee65741fd57e5a0388d70abf3b246d\": container with ID starting with ab7978292402368340713e4f0d991d4cabee65741fd57e5a0388d70abf3b246d not found: ID does not exist"
Oct 30 23:45:51.545534 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:51.545503    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j"]
Oct 30 23:45:51.549714 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:51.549680    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-26ce6b-6fdd76fc6f-pdl4j"]
Oct 30 23:45:52.298044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:45:52.298015    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33314a36-830a-4395-8f74-89227b549913" path="/var/lib/kubelet/pods/33314a36-830a-4395-8f74-89227b549913/volumes"
Oct 30 23:46:02.070015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:02.069972    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k"]
Oct 30 23:46:02.119415 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:02.119376    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-989a4f-ww96k"]
Oct 30 23:46:02.305715 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:02.305682    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="984223ba-f0b0-483c-9587-e2d6420dcfa0" path="/var/lib/kubelet/pods/984223ba-f0b0-483c-9587-e2d6420dcfa0/volumes"
Oct 30 23:46:03.174925 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:03.174889    5078 scope.go:122] "RemoveContainer" containerID="fa3563aef42b1a9936dbf6e5074a81ee8ae3ea932544b0e655c9823907e5be7f"
Oct 30 23:46:03.184515 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:03.184495    5078 scope.go:122] "RemoveContainer" containerID="262afe6c9f0a38e124ba0fed3e5d8e9d711f4392bb9ba2064a5cbc7ed416f5d6"
Oct 30 23:46:03.193453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:03.193435    5078 scope.go:122] "RemoveContainer" containerID="705daa61dd248037ce82a1c729786e1c6711057ee49128d473a1e938c066cb9c"
Oct 30 23:46:13.417428 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:13.417393    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"]
Oct 30 23:46:13.417971 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:13.417577    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp" podUID="313b3ad0-1127-40c2-85b3-4a1555935650" containerName="istio-proxy" containerID="cri-o://2a14e1ae6108ae85129fd8a201c3c45016208cdcde970af0cedd71c52e38c866" gracePeriod=30
Oct 30 23:46:13.429373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:13.429346    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"]
Oct 30 23:46:13.429535 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:13.429516    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467" podUID="7ceff1f9-9fc7-4a93-9619-1605386e0b64" containerName="istio-proxy" containerID="cri-o://f99911c9c866e276cdf24ea346b7cc8d1008c6a5e57436b62b331221a5f4eee1" gracePeriod=30
Oct 30 23:46:13.455274 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:13.455249    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"]
Oct 30 23:46:13.455465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:13.455447    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn" podUID="dc5a135a-f306-48a0-929e-3d9e44f97e1f" containerName="istio-proxy" containerID="cri-o://b45665448e67bd7b57a37bb5c96d6b897366b977a22fdfdbd41f87ca0c6d62dd" gracePeriod=30
Oct 30 23:46:17.482102 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:17.482063    5078 patch_prober.go:29] interesting pod/e2e-mt-55666687-openshift-default-5748f754cc-77jfn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:46:17.482535 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:17.482117    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn" podUID="dc5a135a-f306-48a0-929e-3d9e44f97e1f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:46:17.591272 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:46:17.591229    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:46:18.565702 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.565658    5078 generic.go:356] "Generic (PLEG): container finished" podID="7ceff1f9-9fc7-4a93-9619-1605386e0b64" containerID="f99911c9c866e276cdf24ea346b7cc8d1008c6a5e57436b62b331221a5f4eee1" exitCode=0
Oct 30 23:46:18.565702 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.565674    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467" event={"ID":"7ceff1f9-9fc7-4a93-9619-1605386e0b64","Type":"ContainerDied","Data":"f99911c9c866e276cdf24ea346b7cc8d1008c6a5e57436b62b331221a5f4eee1"}
Oct 30 23:46:18.568350 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.568319    5078 generic.go:356] "Generic (PLEG): container finished" podID="313b3ad0-1127-40c2-85b3-4a1555935650" containerID="2a14e1ae6108ae85129fd8a201c3c45016208cdcde970af0cedd71c52e38c866" exitCode=0
Oct 30 23:46:18.568490 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.568401    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp" event={"ID":"313b3ad0-1127-40c2-85b3-4a1555935650","Type":"ContainerDied","Data":"2a14e1ae6108ae85129fd8a201c3c45016208cdcde970af0cedd71c52e38c866"}
Oct 30 23:46:18.734970 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.734944    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:46:18.762207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.762177    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:46:18.790430 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.790406    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:46:18.798580 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.798560    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istiod-ca-cert\") pod \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") "
Oct 30 23:46:18.798704 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.798688    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-envoy\") pod \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") "
Oct 30 23:46:18.798747 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.798737    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-workload-socket\") pod \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") "
Oct 30 23:46:18.798823 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.798811    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-credential-socket\") pod \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") "
Oct 30 23:46:18.798877 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.798839    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-data\") pod \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") "
Oct 30 23:46:18.798958 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.798866    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istiod-ca-cert" pod "dc5a135a-f306-48a0-929e-3d9e44f97e1f" (UID: "dc5a135a-f306-48a0-929e-3d9e44f97e1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:46:18.798958 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.798875    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-podinfo\") pod \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") "
Oct 30 23:46:18.798958 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.798949    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-workload-certs\") pod \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") "
Oct 30 23:46:18.799121 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.799069    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-credential-socket" pod "dc5a135a-f306-48a0-929e-3d9e44f97e1f" (UID: "dc5a135a-f306-48a0-929e-3d9e44f97e1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:18.799121 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.799085    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-workload-socket" pod "dc5a135a-f306-48a0-929e-3d9e44f97e1f" (UID: "dc5a135a-f306-48a0-929e-3d9e44f97e1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:18.799121 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.799098    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-data" pod "dc5a135a-f306-48a0-929e-3d9e44f97e1f" (UID: "dc5a135a-f306-48a0-929e-3d9e44f97e1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:18.799270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.799142    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-token\" (UniqueName: \"kubernetes.io/projected/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-token\") pod \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") "
Oct 30 23:46:18.799270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.799192    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5a135a-f306-48a0-929e-3d9e44f97e1f-kube-api-access-lfqpv\" (UniqueName: \"kubernetes.io/projected/dc5a135a-f306-48a0-929e-3d9e44f97e1f-kube-api-access-lfqpv\") pod \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\" (UID: \"dc5a135a-f306-48a0-929e-3d9e44f97e1f\") "
Oct 30 23:46:18.799392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.799281    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-workload-certs" pod "dc5a135a-f306-48a0-929e-3d9e44f97e1f" (UID: "dc5a135a-f306-48a0-929e-3d9e44f97e1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:18.799738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.799718    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:18.799738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.799737    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:18.799738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.799746    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:18.799738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.799754    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:18.799738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.799762    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:18.801030 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.801006    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-envoy" pod "dc5a135a-f306-48a0-929e-3d9e44f97e1f" (UID: "dc5a135a-f306-48a0-929e-3d9e44f97e1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:18.801431 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.801415    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-token" pod "dc5a135a-f306-48a0-929e-3d9e44f97e1f" (UID: "dc5a135a-f306-48a0-929e-3d9e44f97e1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:46:18.801770 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.801742    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-podinfo" pod "dc5a135a-f306-48a0-929e-3d9e44f97e1f" (UID: "dc5a135a-f306-48a0-929e-3d9e44f97e1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:46:18.801770 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.801754    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5a135a-f306-48a0-929e-3d9e44f97e1f-kube-api-access-lfqpv" pod "dc5a135a-f306-48a0-929e-3d9e44f97e1f" (UID: "dc5a135a-f306-48a0-929e-3d9e44f97e1f"). InnerVolumeSpecName "kube-api-access-lfqpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:46:18.900787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.900513    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-data\") pod \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") "
Oct 30 23:46:18.900787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.900585    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-credential-socket\") pod \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") "
Oct 30 23:46:18.900787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.900705    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-istio-envoy\") pod \"313b3ad0-1127-40c2-85b3-4a1555935650\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") "
Oct 30 23:46:18.900787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.900779    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ceff1f9-9fc7-4a93-9619-1605386e0b64-kube-api-access-6d729\" (UniqueName: \"kubernetes.io/projected/7ceff1f9-9fc7-4a93-9619-1605386e0b64-kube-api-access-6d729\") pod \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") "
Oct 30 23:46:18.901149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.900827    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-envoy\") pod \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") "
Oct 30 23:46:18.901149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.900855    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/313b3ad0-1127-40c2-85b3-4a1555935650-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313b3ad0-1127-40c2-85b3-4a1555935650-istiod-ca-cert\") pod \"313b3ad0-1127-40c2-85b3-4a1555935650\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") "
Oct 30 23:46:18.901149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.900879    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istiod-ca-cert\") pod \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") "
Oct 30 23:46:18.901149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.900911    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/313b3ad0-1127-40c2-85b3-4a1555935650-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313b3ad0-1127-40c2-85b3-4a1555935650-istio-podinfo\") pod \"313b3ad0-1127-40c2-85b3-4a1555935650\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") "
Oct 30 23:46:18.901149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.900947    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313b3ad0-1127-40c2-85b3-4a1555935650-istio-token\" (UniqueName: \"kubernetes.io/projected/313b3ad0-1127-40c2-85b3-4a1555935650-istio-token\") pod \"313b3ad0-1127-40c2-85b3-4a1555935650\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") "
Oct 30 23:46:18.901149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.900978    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-credential-socket\") pod \"313b3ad0-1127-40c2-85b3-4a1555935650\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") "
Oct 30 23:46:18.901149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.901084    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-workload-socket\") pod \"313b3ad0-1127-40c2-85b3-4a1555935650\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") "
Oct 30 23:46:18.901149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.901121    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-workload-certs\") pod \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") "
Oct 30 23:46:18.901149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.901149    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313b3ad0-1127-40c2-85b3-4a1555935650-kube-api-access-6kwgz\" (UniqueName: \"kubernetes.io/projected/313b3ad0-1127-40c2-85b3-4a1555935650-kube-api-access-6kwgz\") pod \"313b3ad0-1127-40c2-85b3-4a1555935650\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") "
Oct 30 23:46:18.901641 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.901188    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-podinfo\") pod \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") "
Oct 30 23:46:18.901641 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.901267    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-workload-socket\") pod \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") "
Oct 30 23:46:18.901641 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.901320    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-token\" (UniqueName: \"kubernetes.io/projected/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-token\") pod \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\" (UID: \"7ceff1f9-9fc7-4a93-9619-1605386e0b64\") "
Oct 30 23:46:18.901641 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.901368    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-workload-certs\") pod \"313b3ad0-1127-40c2-85b3-4a1555935650\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") "
Oct 30 23:46:18.901641 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.901402    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-istio-data\") pod \"313b3ad0-1127-40c2-85b3-4a1555935650\" (UID: \"313b3ad0-1127-40c2-85b3-4a1555935650\") "
Oct 30 23:46:18.901641 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.901490    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istiod-ca-cert" pod "7ceff1f9-9fc7-4a93-9619-1605386e0b64" (UID: "7ceff1f9-9fc7-4a93-9619-1605386e0b64"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:46:18.901980 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.901749    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-istio-data" pod "313b3ad0-1127-40c2-85b3-4a1555935650" (UID: "313b3ad0-1127-40c2-85b3-4a1555935650"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:18.901980 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.901758    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-credential-socket" pod "7ceff1f9-9fc7-4a93-9619-1605386e0b64" (UID: "7ceff1f9-9fc7-4a93-9619-1605386e0b64"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:18.902827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.902164    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-data" pod "7ceff1f9-9fc7-4a93-9619-1605386e0b64" (UID: "7ceff1f9-9fc7-4a93-9619-1605386e0b64"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:18.902827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.902259    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfqpv\" (UniqueName: \"kubernetes.io/projected/dc5a135a-f306-48a0-929e-3d9e44f97e1f-kube-api-access-lfqpv\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:18.902827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.902284    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:18.902827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.902300    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:18.902827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.902314    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:18.902827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.902331    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:18.902827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.902347    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:18.902827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.902362    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc5a135a-f306-48a0-929e-3d9e44f97e1f-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:18.902827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.902549    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-workload-socket" pod "7ceff1f9-9fc7-4a93-9619-1605386e0b64" (UID: "7ceff1f9-9fc7-4a93-9619-1605386e0b64"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:18.903500 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.903311    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-workload-socket" pod "313b3ad0-1127-40c2-85b3-4a1555935650" (UID: "313b3ad0-1127-40c2-85b3-4a1555935650"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:18.903500 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.903365    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-workload-certs" pod "313b3ad0-1127-40c2-85b3-4a1555935650" (UID: "313b3ad0-1127-40c2-85b3-4a1555935650"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:18.903664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.903490    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-istio-envoy" pod "313b3ad0-1127-40c2-85b3-4a1555935650" (UID: "313b3ad0-1127-40c2-85b3-4a1555935650"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:18.903664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.903576    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-credential-socket" pod "313b3ad0-1127-40c2-85b3-4a1555935650" (UID: "313b3ad0-1127-40c2-85b3-4a1555935650"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:18.903816 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.903781    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-workload-certs" pod "7ceff1f9-9fc7-4a93-9619-1605386e0b64" (UID: "7ceff1f9-9fc7-4a93-9619-1605386e0b64"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:18.903983 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.903879    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/313b3ad0-1127-40c2-85b3-4a1555935650-istiod-ca-cert" pod "313b3ad0-1127-40c2-85b3-4a1555935650" (UID: "313b3ad0-1127-40c2-85b3-4a1555935650"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:46:18.904637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.904461    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313b3ad0-1127-40c2-85b3-4a1555935650-kube-api-access-6kwgz" pod "313b3ad0-1127-40c2-85b3-4a1555935650" (UID: "313b3ad0-1127-40c2-85b3-4a1555935650"). InnerVolumeSpecName "kube-api-access-6kwgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:46:18.904637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.904573    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/313b3ad0-1127-40c2-85b3-4a1555935650-istio-podinfo" pod "313b3ad0-1127-40c2-85b3-4a1555935650" (UID: "313b3ad0-1127-40c2-85b3-4a1555935650"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:46:18.906198 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.906173    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ceff1f9-9fc7-4a93-9619-1605386e0b64-kube-api-access-6d729" pod "7ceff1f9-9fc7-4a93-9619-1605386e0b64" (UID: "7ceff1f9-9fc7-4a93-9619-1605386e0b64"). InnerVolumeSpecName "kube-api-access-6d729". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:46:18.906305 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.906265    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-token" pod "7ceff1f9-9fc7-4a93-9619-1605386e0b64" (UID: "7ceff1f9-9fc7-4a93-9619-1605386e0b64"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:46:18.906376 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.906316    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-podinfo" pod "7ceff1f9-9fc7-4a93-9619-1605386e0b64" (UID: "7ceff1f9-9fc7-4a93-9619-1605386e0b64"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:46:18.907348 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.907325    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313b3ad0-1127-40c2-85b3-4a1555935650-istio-token" pod "313b3ad0-1127-40c2-85b3-4a1555935650" (UID: "313b3ad0-1127-40c2-85b3-4a1555935650"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:46:18.907643 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:18.907623    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-envoy" pod "7ceff1f9-9fc7-4a93-9619-1605386e0b64" (UID: "7ceff1f9-9fc7-4a93-9619-1605386e0b64"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:19.011541 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.011452    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:19.011541 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.011481    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:19.011541 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.011492    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d729\" (UniqueName: \"kubernetes.io/projected/7ceff1f9-9fc7-4a93-9619-1605386e0b64-kube-api-access-6d729\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:19.011541 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.011501    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:19.011541 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.011511    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/313b3ad0-1127-40c2-85b3-4a1555935650-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:19.011541 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.011519    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/313b3ad0-1127-40c2-85b3-4a1555935650-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:19.011541 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.011527    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/313b3ad0-1127-40c2-85b3-4a1555935650-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:19.011541 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.011544    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:19.011946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.011553    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:19.011946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.011562    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:19.011946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.011570    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kwgz\" (UniqueName: \"kubernetes.io/projected/313b3ad0-1127-40c2-85b3-4a1555935650-kube-api-access-6kwgz\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:19.011946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.011578    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:19.011946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.011586    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ceff1f9-9fc7-4a93-9619-1605386e0b64-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:19.011946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.011782    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ceff1f9-9fc7-4a93-9619-1605386e0b64-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:19.011946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.011802    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/313b3ad0-1127-40c2-85b3-4a1555935650-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:19.576466 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.576434    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"
Oct 30 23:46:19.576466 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.576451    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467" event={"ID":"7ceff1f9-9fc7-4a93-9619-1605386e0b64","Type":"ContainerDied","Data":"5da5f5b6ef02458ca1197803db97770100cc6f4661201af041df9e5a7987f7b7"}
Oct 30 23:46:19.577043 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.576517    5078 scope.go:122] "RemoveContainer" containerID="f99911c9c866e276cdf24ea346b7cc8d1008c6a5e57436b62b331221a5f4eee1"
Oct 30 23:46:19.578703 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.578675    5078 generic.go:356] "Generic (PLEG): container finished" podID="dc5a135a-f306-48a0-929e-3d9e44f97e1f" containerID="b45665448e67bd7b57a37bb5c96d6b897366b977a22fdfdbd41f87ca0c6d62dd" exitCode=0
Oct 30 23:46:19.578868 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.578749    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"
Oct 30 23:46:19.578868 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.578760    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn" event={"ID":"dc5a135a-f306-48a0-929e-3d9e44f97e1f","Type":"ContainerDied","Data":"b45665448e67bd7b57a37bb5c96d6b897366b977a22fdfdbd41f87ca0c6d62dd"}
Oct 30 23:46:19.579013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.578871    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn" event={"ID":"dc5a135a-f306-48a0-929e-3d9e44f97e1f","Type":"ContainerDied","Data":"c73e96664c27b91a2590677cbe13fe98fd4e196eff6d1641fbbf609923bd7244"}
Oct 30 23:46:19.580438 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.580420    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"
Oct 30 23:46:19.580560 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.580443    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp" event={"ID":"313b3ad0-1127-40c2-85b3-4a1555935650","Type":"ContainerDied","Data":"ed8db85eba9d810936ec40b6b1070ad480fbf7a1d2d6ec91862876d9a48f84d5"}
Oct 30 23:46:19.588328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.588312    5078 scope.go:122] "RemoveContainer" containerID="b45665448e67bd7b57a37bb5c96d6b897366b977a22fdfdbd41f87ca0c6d62dd"
Oct 30 23:46:19.598323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.598298    5078 scope.go:122] "RemoveContainer" containerID="a97256900f72473dae2680bdd03f16d45793d684590c300bfe40fe3927a38e46"
Oct 30 23:46:19.607119 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.607103    5078 scope.go:122] "RemoveContainer" containerID="b45665448e67bd7b57a37bb5c96d6b897366b977a22fdfdbd41f87ca0c6d62dd"
Oct 30 23:46:19.607418 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:46:19.607394    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45665448e67bd7b57a37bb5c96d6b897366b977a22fdfdbd41f87ca0c6d62dd\": container with ID starting with b45665448e67bd7b57a37bb5c96d6b897366b977a22fdfdbd41f87ca0c6d62dd not found: ID does not exist" containerID="b45665448e67bd7b57a37bb5c96d6b897366b977a22fdfdbd41f87ca0c6d62dd"
Oct 30 23:46:19.607470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.607431    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45665448e67bd7b57a37bb5c96d6b897366b977a22fdfdbd41f87ca0c6d62dd"} err="failed to get container status \"b45665448e67bd7b57a37bb5c96d6b897366b977a22fdfdbd41f87ca0c6d62dd\": rpc error: code = NotFound desc = could not find container \"b45665448e67bd7b57a37bb5c96d6b897366b977a22fdfdbd41f87ca0c6d62dd\": container with ID starting with b45665448e67bd7b57a37bb5c96d6b897366b977a22fdfdbd41f87ca0c6d62dd not found: ID does not exist"
Oct 30 23:46:19.607470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.607455    5078 scope.go:122] "RemoveContainer" containerID="a97256900f72473dae2680bdd03f16d45793d684590c300bfe40fe3927a38e46"
Oct 30 23:46:19.607782 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:46:19.607759    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a97256900f72473dae2680bdd03f16d45793d684590c300bfe40fe3927a38e46\": container with ID starting with a97256900f72473dae2680bdd03f16d45793d684590c300bfe40fe3927a38e46 not found: ID does not exist" containerID="a97256900f72473dae2680bdd03f16d45793d684590c300bfe40fe3927a38e46"
Oct 30 23:46:19.607865 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.607794    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a97256900f72473dae2680bdd03f16d45793d684590c300bfe40fe3927a38e46"} err="failed to get container status \"a97256900f72473dae2680bdd03f16d45793d684590c300bfe40fe3927a38e46\": rpc error: code = NotFound desc = could not find container \"a97256900f72473dae2680bdd03f16d45793d684590c300bfe40fe3927a38e46\": container with ID starting with a97256900f72473dae2680bdd03f16d45793d684590c300bfe40fe3927a38e46 not found: ID does not exist"
Oct 30 23:46:19.607865 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.607815    5078 scope.go:122] "RemoveContainer" containerID="2a14e1ae6108ae85129fd8a201c3c45016208cdcde970af0cedd71c52e38c866"
Oct 30 23:46:19.742869 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.742785    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"]
Oct 30 23:46:19.792177 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.792143    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-99qzp"]
Oct 30 23:46:19.854973 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.854944    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"]
Oct 30 23:46:19.858869 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.858844    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-77jfn"]
Oct 30 23:46:19.967409 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.967377    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"]
Oct 30 23:46:19.970792 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:19.970767    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-55666687-openshift-default-5748f754cc-sp467"]
Oct 30 23:46:20.298206 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:20.298176    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="313b3ad0-1127-40c2-85b3-4a1555935650" path="/var/lib/kubelet/pods/313b3ad0-1127-40c2-85b3-4a1555935650/volumes"
Oct 30 23:46:20.298671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:20.298657    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ceff1f9-9fc7-4a93-9619-1605386e0b64" path="/var/lib/kubelet/pods/7ceff1f9-9fc7-4a93-9619-1605386e0b64/volumes"
Oct 30 23:46:20.299057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:20.299045    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc5a135a-f306-48a0-929e-3d9e44f97e1f" path="/var/lib/kubelet/pods/dc5a135a-f306-48a0-929e-3d9e44f97e1f/volumes"
Oct 30 23:46:32.427176 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427142    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"]
Oct 30 23:46:32.427637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427579    5078 cpu_manager.go:414] "containerMap: removing container" podUID="dc5a135a-f306-48a0-929e-3d9e44f97e1f" containerName="istio-proxy"
Oct 30 23:46:32.427637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427590    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc5a135a-f306-48a0-929e-3d9e44f97e1f" containerName="istio-proxy"
Oct 30 23:46:32.427637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427618    5078 cpu_manager.go:414] "containerMap: removing container" podUID="dc5a135a-f306-48a0-929e-3d9e44f97e1f" containerName="istio-proxy"
Oct 30 23:46:32.427637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427623    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc5a135a-f306-48a0-929e-3d9e44f97e1f" containerName="istio-proxy"
Oct 30 23:46:32.427637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427633    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d365bd08-d827-41b2-83ba-c244d745c64e" containerName="revoke-keys"
Oct 30 23:46:32.427637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427638    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d365bd08-d827-41b2-83ba-c244d745c64e" containerName="revoke-keys"
Oct 30 23:46:32.427855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427646    5078 cpu_manager.go:414] "containerMap: removing container" podUID="eb7f4356-b694-4ebb-a65e-8f7782591aea" containerName="istio-proxy"
Oct 30 23:46:32.427855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427651    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb7f4356-b694-4ebb-a65e-8f7782591aea" containerName="istio-proxy"
Oct 30 23:46:32.427855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427658    5078 cpu_manager.go:414] "containerMap: removing container" podUID="33314a36-830a-4395-8f74-89227b549913" containerName="payload-processing"
Oct 30 23:46:32.427855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427662    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33314a36-830a-4395-8f74-89227b549913" containerName="payload-processing"
Oct 30 23:46:32.427855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427681    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7ceff1f9-9fc7-4a93-9619-1605386e0b64" containerName="istio-proxy"
Oct 30 23:46:32.427855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427685    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ceff1f9-9fc7-4a93-9619-1605386e0b64" containerName="istio-proxy"
Oct 30 23:46:32.427855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427693    5078 cpu_manager.go:414] "containerMap: removing container" podUID="313b3ad0-1127-40c2-85b3-4a1555935650" containerName="istio-proxy"
Oct 30 23:46:32.427855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427697    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313b3ad0-1127-40c2-85b3-4a1555935650" containerName="istio-proxy"
Oct 30 23:46:32.427855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427785    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc5a135a-f306-48a0-929e-3d9e44f97e1f" containerName="istio-proxy"
Oct 30 23:46:32.427855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427795    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d365bd08-d827-41b2-83ba-c244d745c64e" containerName="revoke-keys"
Oct 30 23:46:32.427855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427801    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb7f4356-b694-4ebb-a65e-8f7782591aea" containerName="istio-proxy"
Oct 30 23:46:32.427855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427805    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="33314a36-830a-4395-8f74-89227b549913" containerName="payload-processing"
Oct 30 23:46:32.427855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427814    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ceff1f9-9fc7-4a93-9619-1605386e0b64" containerName="istio-proxy"
Oct 30 23:46:32.427855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.427819    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="313b3ad0-1127-40c2-85b3-4a1555935650" containerName="istio-proxy"
Oct 30 23:46:32.432937 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.432916    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.517864 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.517828    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7c36d6-openshift-default-dockercfg-mk4fc\""
Oct 30 23:46:32.620689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.600670    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cffmb\" (UniqueName: \"kubernetes.io/projected/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-kube-api-access-cffmb\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.620689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.600748    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-envoy\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.620689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.600781    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-token\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.620689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.600844    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-data\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.620689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.600914    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-credential-socket\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.620689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.600965    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-podinfo\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.620689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.601008    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istiod-ca-cert\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.620689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.601065    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-workload-socket\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.620689 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.601091    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-workload-certs\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.629678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.629645    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"]
Oct 30 23:46:32.702098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.702009    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cffmb\" (UniqueName: \"kubernetes.io/projected/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-kube-api-access-cffmb\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.702098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.702066    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-envoy\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.702098 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.702096    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-token\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.702385 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.702152    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-data\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.702385 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.702213    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-credential-socket\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.702385 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.702260    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-podinfo\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.702385 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.702298    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istiod-ca-cert\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.702385 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.702346    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-workload-socket\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.702385 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.702370    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-workload-certs\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.702795 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.702772    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-workload-certs\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.703228 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.703150    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-credential-socket\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.703300 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.703285    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-workload-socket\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.703924 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.703898    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-data\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.704031 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.703946    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istiod-ca-cert\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.706004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.705978    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-envoy\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.706143 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.706124    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-podinfo\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.784566 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.784539    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-token\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.785143 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.785117    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cffmb\" (UniqueName: \"kubernetes.io/projected/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-kube-api-access-cffmb\") pod \"e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") " pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:32.927440 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.927405    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr"]
Oct 30 23:46:32.929907 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:32.929880    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-057303-7jgjr"]
Oct 30 23:46:33.073624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:33.072842    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:33.506036 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:33.506012    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"]
Oct 30 23:46:33.507918 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:46:33.507886    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1e8c37_0f6c_4a6e_b877_43a235878ca8.slice/crio-2aaa68bb4e6dda23842df319ce97a22945715fe0e127b5ff1728481763c59037 WatchSource:0}: Error finding container 2aaa68bb4e6dda23842df319ce97a22945715fe0e127b5ff1728481763c59037: Status 404 returned error can't find the container with id 2aaa68bb4e6dda23842df319ce97a22945715fe0e127b5ff1728481763c59037
Oct 30 23:46:33.510283 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:33.510249    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:33.510374 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:33.510317    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:33.510374 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:33.510348    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:33.778539 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:33.777428    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz" event={"ID":"fc1e8c37-0f6c-4a6e-b877-43a235878ca8","Type":"ContainerStarted","Data":"e76df04832f9ecc5ef738d9f9ce5aaee769e833ab8f4ea1f09235962e2ab58e0"}
Oct 30 23:46:33.778539 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:33.777493    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz" event={"ID":"fc1e8c37-0f6c-4a6e-b877-43a235878ca8","Type":"ContainerStarted","Data":"2aaa68bb4e6dda23842df319ce97a22945715fe0e127b5ff1728481763c59037"}
Oct 30 23:46:33.900155 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:33.900096    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz" podStartSLOduration=1.9000775330000002 podStartE2EDuration="1.900077533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:33.898639219 +0000 UTC m=+2379.786545573" watchObservedRunningTime="2026-10-30 23:46:33.900077533 +0000 UTC m=+2379.787983890"
Oct 30 23:46:34.076650 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:34.076252    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:34.090970 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:34.090937    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:34.297794 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:34.297765    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501" path="/var/lib/kubelet/pods/7c4e5d9c-27d0-4dda-8c8e-93f6ac9b6501/volumes"
Oct 30 23:46:34.783916 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:34.783886    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:34.785033 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:34.785015    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:46:35.326821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.326791    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"]
Oct 30 23:46:35.327279 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.327266    5078 cpu_manager.go:414] "containerMap: removing container" podUID="eb7f4356-b694-4ebb-a65e-8f7782591aea" containerName="istio-proxy"
Oct 30 23:46:35.327333 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.327280    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb7f4356-b694-4ebb-a65e-8f7782591aea" containerName="istio-proxy"
Oct 30 23:46:35.327374 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.327349    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc5a135a-f306-48a0-929e-3d9e44f97e1f" containerName="istio-proxy"
Oct 30 23:46:35.327374 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.327361    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb7f4356-b694-4ebb-a65e-8f7782591aea" containerName="istio-proxy"
Oct 30 23:46:35.330488 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.330472    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.420637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.420582    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"]
Oct 30 23:46:35.440191 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.440149    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.440395 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.440215    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.440395 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.440363    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.440512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.440457    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.440551 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.440520    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.440625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.440560    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.440679 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.440647    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v7vp\" (UniqueName: \"kubernetes.io/projected/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-kube-api-access-8v7vp\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.440728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.440695    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.440884 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.440862    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.542906 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.542866    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.543207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.543186    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.543465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.543432    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.543582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.543500    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.543582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.543562    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.543693 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.543627    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.543693 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.543680    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.543771 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.543722    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v7vp\" (UniqueName: \"kubernetes.io/projected/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-kube-api-access-8v7vp\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.543824 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.543783    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.545339 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.544583    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.545339 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.544908    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.545339 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.545055    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.545339 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.545076    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.545968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.545941    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.547293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.547273    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.549212 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.549187    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.600663 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.600568    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.602541 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.602514    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v7vp\" (UniqueName: \"kubernetes.io/projected/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-kube-api-access-8v7vp\") pod \"maas-default-gateway-openshift-default-5df48fcc99-wzqb8\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:35.645099 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:35.645057    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:36.077951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.077922    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"]
Oct 30 23:46:36.079802 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:46:36.079775    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f8595d_81df_4243_bb4c_7afaa5f3ca26.slice/crio-1744fcaf31222552be557e397177cc6a4a8e2954560b75609659e1fae1da7e35 WatchSource:0}: Error finding container 1744fcaf31222552be557e397177cc6a4a8e2954560b75609659e1fae1da7e35: Status 404 returned error can't find the container with id 1744fcaf31222552be557e397177cc6a4a8e2954560b75609659e1fae1da7e35
Oct 30 23:46:36.082560 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.082529    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:36.082676 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.082618    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:36.082676 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.082655    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:36.256831 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.256795    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"]
Oct 30 23:46:36.260759 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.260729    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.325430 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.325402    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"]
Oct 30 23:46:36.354363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.354251    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-workload-certs\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.354363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.354319    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istiod-ca-cert\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.354685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.354427    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-data\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.354685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.354483    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-podinfo\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.354685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.354590    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-token\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.354685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.354644    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-workload-socket\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.354953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.354785    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-credential-socket\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.354953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.354820    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-envoy\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.354953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.354848    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t44k9\" (UniqueName: \"kubernetes.io/projected/abd64869-cfaf-4fec-b73d-90bfb3cc8819-kube-api-access-t44k9\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.456009 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.455970    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-credential-socket\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.456009 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.456019    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-envoy\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.456309 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.456144    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t44k9\" (UniqueName: \"kubernetes.io/projected/abd64869-cfaf-4fec-b73d-90bfb3cc8819-kube-api-access-t44k9\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.456309 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.456273    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-workload-certs\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.456431 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.456314    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istiod-ca-cert\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.456431 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.456342    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-data\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.456431 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.456377    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-podinfo\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.456637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.456456    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-credential-socket\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.456637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.456473    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-token\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.456637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.456508    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-workload-socket\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.456637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.456578    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-workload-certs\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.456909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.456884    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-workload-socket\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.457101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.457081    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istiod-ca-cert\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.457201 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.457096    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-data\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.459075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.459047    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-envoy\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.459190 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.459083    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-podinfo\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.529061 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.529026    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t44k9\" (UniqueName: \"kubernetes.io/projected/abd64869-cfaf-4fec-b73d-90bfb3cc8819-kube-api-access-t44k9\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.532728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.532700    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-token\") pod \"e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") " pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.577737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.577708    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:36.804414 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.804370    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8" event={"ID":"c8f8595d-81df-4243-bb4c-7afaa5f3ca26","Type":"ContainerStarted","Data":"fc855c6495858b05ef58491bd002195c0477c5752fe6b3657c1e800cd5a500a2"}
Oct 30 23:46:36.804660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.804435    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8" event={"ID":"c8f8595d-81df-4243-bb4c-7afaa5f3ca26","Type":"ContainerStarted","Data":"1744fcaf31222552be557e397177cc6a4a8e2954560b75609659e1fae1da7e35"}
Oct 30 23:46:36.928277 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:36.928165    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8" podStartSLOduration=1.928150442 podStartE2EDuration="1.928150442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:36.924716085 +0000 UTC m=+2382.812622438" watchObservedRunningTime="2026-10-30 23:46:36.928150442 +0000 UTC m=+2382.816056795"
Oct 30 23:46:37.015224 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:37.015191    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"]
Oct 30 23:46:37.017014 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:46:37.016987    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd64869_cfaf_4fec_b73d_90bfb3cc8819.slice/crio-e17f2d25c84439a9da4ca3d0a60a7ae8d6f46beb08f6ca02762e0e05b84b0f38 WatchSource:0}: Error finding container e17f2d25c84439a9da4ca3d0a60a7ae8d6f46beb08f6ca02762e0e05b84b0f38: Status 404 returned error can't find the container with id e17f2d25c84439a9da4ca3d0a60a7ae8d6f46beb08f6ca02762e0e05b84b0f38
Oct 30 23:46:37.019806 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:37.019772    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:37.019920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:37.019839    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:37.019920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:37.019892    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:37.658511 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:37.658482    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:37.672697 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:37.671804    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wzqb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 23:46:37.672697 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:37.671866    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8" podUID="c8f8595d-81df-4243-bb4c-7afaa5f3ca26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 23:46:37.827165 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:37.827126    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469" event={"ID":"abd64869-cfaf-4fec-b73d-90bfb3cc8819","Type":"ContainerStarted","Data":"3808003d13cdecb9a8f48485a324d9a949afd13f102878e632c3fc058bfd49fe"}
Oct 30 23:46:37.827365 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:37.827194    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469" event={"ID":"abd64869-cfaf-4fec-b73d-90bfb3cc8819","Type":"ContainerStarted","Data":"e17f2d25c84439a9da4ca3d0a60a7ae8d6f46beb08f6ca02762e0e05b84b0f38"}
Oct 30 23:46:37.951761 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:37.951561    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469" podStartSLOduration=1.95154038 podStartE2EDuration="1.95154038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:37.950199316 +0000 UTC m=+2383.838105709" watchObservedRunningTime="2026-10-30 23:46:37.95154038 +0000 UTC m=+2383.839446732"
Oct 30 23:46:38.586622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:38.583408    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:38.586622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:38.583588    5078 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 23:46:38.586622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:38.583651    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469" podUID="abd64869-cfaf-4fec-b73d-90bfb3cc8819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 23:46:38.586622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:38.585731    5078 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:46:38.586622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:38.585782    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c" podUID="412494b5-417a-488e-9ce4-9771d6458325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:46:38.645459 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:38.645425    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wzqb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 23:46:38.645646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:38.645484    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8" podUID="c8f8595d-81df-4243-bb4c-7afaa5f3ca26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 23:46:39.592625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:39.592084    5078 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 23:46:39.592625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:39.592150    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469" podUID="abd64869-cfaf-4fec-b73d-90bfb3cc8819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 23:46:39.646198 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:39.646151    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wzqb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 23:46:39.646368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:39.646221    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8" podUID="c8f8595d-81df-4243-bb4c-7afaa5f3ca26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 23:46:40.150624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:40.150536    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89"]
Oct 30 23:46:40.194344 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:40.194257    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-26ce6b-w6j89"]
Oct 30 23:46:40.327739 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:40.327699    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d365bd08-d827-41b2-83ba-c244d745c64e" path="/var/lib/kubelet/pods/d365bd08-d827-41b2-83ba-c244d745c64e/volumes"
Oct 30 23:46:40.580995 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:40.580959    5078 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 23:46:40.581307 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:40.581279    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469" podUID="abd64869-cfaf-4fec-b73d-90bfb3cc8819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 23:46:40.645720 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:40.645690    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wzqb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 23:46:40.646127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:40.645743    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8" podUID="c8f8595d-81df-4243-bb4c-7afaa5f3ca26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 23:46:40.852680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:40.852650    5078 generic.go:356] "Generic (PLEG): container finished" podID="412494b5-417a-488e-9ce4-9771d6458325" containerID="347067e02bdf480b23781cb99d09e82cd306cecd5979478f5d61d6abc399addb" exitCode=0
Oct 30 23:46:40.852871 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:40.852726    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c" event={"ID":"412494b5-417a-488e-9ce4-9771d6458325","Type":"ContainerDied","Data":"347067e02bdf480b23781cb99d09e82cd306cecd5979478f5d61d6abc399addb"}
Oct 30 23:46:40.853175 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:40.853161    5078 scope.go:122] "RemoveContainer" containerID="347067e02bdf480b23781cb99d09e82cd306cecd5979478f5d61d6abc399addb"
Oct 30 23:46:40.854647 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:40.854625    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:40.854712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:40.854691    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:40.854749 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:40.854720    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:41.108838 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.101657    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"]
Oct 30 23:46:41.108838 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.106237    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.191905 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.191873    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-15b137-openshift-default-dockercfg-594dk\""
Oct 30 23:46:41.265618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.263202    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"]
Oct 30 23:46:41.317979 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.292404    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-envoy\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.317979 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.306724    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-workload-certs\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.317979 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.306845    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-credential-socket\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.317979 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.306889    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-workload-socket\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.317979 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.306957    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-token\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.317979 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.307070    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-data\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.317979 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.307120    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d43311-ddb3-4d7f-901d-025555bc9e37-istiod-ca-cert\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.317979 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.307180    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-podinfo\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.317979 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.307266    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4ttt\" (UniqueName: \"kubernetes.io/projected/d6d43311-ddb3-4d7f-901d-025555bc9e37-kube-api-access-k4ttt\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.437827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.432980    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-envoy\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.437827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.433029    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-workload-certs\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.437827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.433076    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-credential-socket\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.437827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.433109    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-workload-socket\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.437827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.433164    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-token\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.437827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.433249    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-data\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.437827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.433290    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d43311-ddb3-4d7f-901d-025555bc9e37-istiod-ca-cert\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.437827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.433340    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-podinfo\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.437827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.433411    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4ttt\" (UniqueName: \"kubernetes.io/projected/d6d43311-ddb3-4d7f-901d-025555bc9e37-kube-api-access-k4ttt\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.437827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.434155    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-workload-certs\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.437827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.434272    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-data\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.437827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.434742    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-credential-socket\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.437827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.434945    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d43311-ddb3-4d7f-901d-025555bc9e37-istiod-ca-cert\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.437827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.435332    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-workload-socket\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.438732 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.438088    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-podinfo\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.446988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.446544    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-envoy\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.501662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.500374    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4ttt\" (UniqueName: \"kubernetes.io/projected/d6d43311-ddb3-4d7f-901d-025555bc9e37-kube-api-access-k4ttt\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.534623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.534080    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-token\") pod \"e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") " pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.584623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.584551    5078 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 23:46:41.584841 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.584631    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469" podUID="abd64869-cfaf-4fec-b73d-90bfb3cc8819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 23:46:41.646342 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.646292    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wzqb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 23:46:41.646760 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.646363    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8" podUID="c8f8595d-81df-4243-bb4c-7afaa5f3ca26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 23:46:41.678621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.675838    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:41.923395 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:41.923302    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c" event={"ID":"412494b5-417a-488e-9ce4-9771d6458325","Type":"ContainerStarted","Data":"3ffc8952f288a4a2609e8f2f1da8db78733580dd3362178644a5b447184395fe"}
Oct 30 23:46:42.218655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:42.215939    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"]
Oct 30 23:46:42.236851 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:42.236175    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:42.236851 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:42.236279    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:42.236851 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:42.236321    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:42.534016 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:42.531576    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:46:42.534016 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:42.531762    5078 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 23:46:42.534016 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:42.531805    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c" podUID="412494b5-417a-488e-9ce4-9771d6458325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 23:46:42.602622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:42.602081    5078 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 23:46:42.602622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:42.602150    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469" podUID="abd64869-cfaf-4fec-b73d-90bfb3cc8819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 23:46:42.646185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:42.646149    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wzqb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 23:46:42.646473 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:42.646444    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8" podUID="c8f8595d-81df-4243-bb4c-7afaa5f3ca26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 23:46:42.932649 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:42.932617    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm" event={"ID":"d6d43311-ddb3-4d7f-901d-025555bc9e37","Type":"ContainerStarted","Data":"afb312b3d4f255bee0b333827314ef9ab4f8f0002a021337ca8cd2b0360f6e1a"}
Oct 30 23:46:42.932892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:42.932671    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm" event={"ID":"d6d43311-ddb3-4d7f-901d-025555bc9e37","Type":"ContainerStarted","Data":"fbbd0dbda3577a75ca38fe9f5e086a23cd4731a3d54c25afcf117ab1891469ee"}
Oct 30 23:46:43.050132 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:43.050071    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm" podStartSLOduration=3.050050226 podStartE2EDuration="3.050050226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:43.049783284 +0000 UTC m=+2388.937689643" watchObservedRunningTime="2026-10-30 23:46:43.050050226 +0000 UTC m=+2388.937956581"
Oct 30 23:46:43.531297 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:43.531256    5078 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 23:46:43.531485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:43.531319    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c" podUID="412494b5-417a-488e-9ce4-9771d6458325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 23:46:43.580620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:43.579933    5078 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused" start-of-body=
Oct 30 23:46:43.580620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:43.579992    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469" podUID="abd64869-cfaf-4fec-b73d-90bfb3cc8819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.212:15021/healthz/ready\": dial tcp 10.133.0.212:15021: connect: connection refused"
Oct 30 23:46:43.646618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:43.645999    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wzqb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 23:46:43.646618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:43.646076    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8" podUID="c8f8595d-81df-4243-bb4c-7afaa5f3ca26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 23:46:43.676787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:43.676744    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:43.780728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:43.780700    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:43.942399 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:43.942355    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:43.943923 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:43.943900    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:46:44.533622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.532743    5078 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 23:46:44.533622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.532804    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c" podUID="412494b5-417a-488e-9ce4-9771d6458325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 23:46:44.534019 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.533644    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv"]
Oct 30 23:46:44.547621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.544476    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv"
Oct 30 23:46:44.588620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.587240    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv"]
Oct 30 23:46:44.599616 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.597383    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7c36d6\""
Oct 30 23:46:44.599616 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.597524    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7c36d6-dockercfg-qqtkb\""
Oct 30 23:46:44.667843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.667810    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-wzqb8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused" start-of-body=
Oct 30 23:46:44.674793 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.674730    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8" podUID="c8f8595d-81df-4243-bb4c-7afaa5f3ca26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.211:15021/healthz/ready\": dial tcp 10.133.0.211:15021: connect: connection refused"
Oct 30 23:46:44.700659 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.685134    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm"]
Oct 30 23:46:44.711660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.702251    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm"
Oct 30 23:46:44.840129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.840081    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv\" (UID: \"b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv"
Oct 30 23:46:44.855225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.851393    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm"]
Oct 30 23:46:44.855225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.854458    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbt6g\" (UniqueName: \"kubernetes.io/projected/b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8-kube-api-access-zbt6g\") pod \"payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv\" (UID: \"b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv"
Oct 30 23:46:44.956612 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.956528    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dt7t\" (UniqueName: \"kubernetes.io/projected/de020d7e-ce62-476f-9ce1-1ecd344f05b8-kube-api-access-4dt7t\") pod \"payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm\" (UID: \"de020d7e-ce62-476f-9ce1-1ecd344f05b8\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm"
Oct 30 23:46:44.956833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.956669    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de020d7e-ce62-476f-9ce1-1ecd344f05b8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm\" (UID: \"de020d7e-ce62-476f-9ce1-1ecd344f05b8\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm"
Oct 30 23:46:44.964628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.956905    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv\" (UID: \"b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv"
Oct 30 23:46:44.964628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.956972    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbt6g\" (UniqueName: \"kubernetes.io/projected/b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8-kube-api-access-zbt6g\") pod \"payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv\" (UID: \"b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv"
Oct 30 23:46:44.964628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:44.958878    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv\" (UID: \"b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv"
Oct 30 23:46:45.015012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.014977    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbt6g\" (UniqueName: \"kubernetes.io/projected/b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8-kube-api-access-zbt6g\") pod \"payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv\" (UID: \"b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv"
Oct 30 23:46:45.017777 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.017750    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:45.018570 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.018544    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:45.019764 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.019742    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:46:45.061614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.061548    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dt7t\" (UniqueName: \"kubernetes.io/projected/de020d7e-ce62-476f-9ce1-1ecd344f05b8-kube-api-access-4dt7t\") pod \"payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm\" (UID: \"de020d7e-ce62-476f-9ce1-1ecd344f05b8\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm"
Oct 30 23:46:45.068765 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.067745    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de020d7e-ce62-476f-9ce1-1ecd344f05b8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm\" (UID: \"de020d7e-ce62-476f-9ce1-1ecd344f05b8\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm"
Oct 30 23:46:45.069056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.068722    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de020d7e-ce62-476f-9ce1-1ecd344f05b8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm\" (UID: \"de020d7e-ce62-476f-9ce1-1ecd344f05b8\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm"
Oct 30 23:46:45.126238 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.126164    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dt7t\" (UniqueName: \"kubernetes.io/projected/de020d7e-ce62-476f-9ce1-1ecd344f05b8-kube-api-access-4dt7t\") pod \"payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm\" (UID: \"de020d7e-ce62-476f-9ce1-1ecd344f05b8\") " pod="openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm"
Oct 30 23:46:45.194858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.191756    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv"
Oct 30 23:46:45.423804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.423511    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm"
Oct 30 23:46:45.536555 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.535913    5078 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 23:46:45.536555 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.535975    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c" podUID="412494b5-417a-488e-9ce4-9771d6458325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 23:46:45.606489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.606441    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv"]
Oct 30 23:46:45.841651 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.841619    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm"]
Oct 30 23:46:45.843063 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:46:45.843035    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde020d7e_ce62_476f_9ce1_1ecd344f05b8.slice/crio-64215e86c5606cfb849015197fe862720ad9907d7fb9513770563dc4c027f56e WatchSource:0}: Error finding container 64215e86c5606cfb849015197fe862720ad9907d7fb9513770563dc4c027f56e: Status 404 returned error can't find the container with id 64215e86c5606cfb849015197fe862720ad9907d7fb9513770563dc4c027f56e
Oct 30 23:46:45.994738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.994646    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv" event={"ID":"b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8","Type":"ContainerStarted","Data":"0f1b58d42637429fef957c46f4df3331a96128ffe244912612fbbeece223ba3a"}
Oct 30 23:46:45.994738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.994725    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv" event={"ID":"b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8","Type":"ContainerStarted","Data":"0abb42fb8a0ce941df706958b669f4a7436aaa8fcd31ba046eefd0c15c1f928d"}
Oct 30 23:46:45.994969 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.994757    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv"
Oct 30 23:46:45.996742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.996714    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm" event={"ID":"de020d7e-ce62-476f-9ce1-1ecd344f05b8","Type":"ContainerStarted","Data":"7cd2dd1e56af7184ef60d8d45fe5816f29872033898cb17051e00a3b31465050"}
Oct 30 23:46:45.996921 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:45.996813    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm" event={"ID":"de020d7e-ce62-476f-9ce1-1ecd344f05b8","Type":"ContainerStarted","Data":"64215e86c5606cfb849015197fe862720ad9907d7fb9513770563dc4c027f56e"}
Oct 30 23:46:46.111014 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:46.110949    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv" podStartSLOduration=2.110929638 podStartE2EDuration="2.110929638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:46.107818291 +0000 UTC m=+2391.995724646" watchObservedRunningTime="2026-10-30 23:46:46.110929638 +0000 UTC m=+2391.998835995"
Oct 30 23:46:46.386625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:46.341762    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm" podStartSLOduration=2.341737567 podStartE2EDuration="2.341737567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:46.289941565 +0000 UTC m=+2392.177847919" watchObservedRunningTime="2026-10-30 23:46:46.341737567 +0000 UTC m=+2392.229643933"
Oct 30 23:46:46.531673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:46.531635    5078 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 23:46:46.531849 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:46.531692    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c" podUID="412494b5-417a-488e-9ce4-9771d6458325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 23:46:46.545060 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:46.545029    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:46.546852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:46.546824    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:46.547834 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:46.547801    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:46:47.003203 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:47.003167    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm"
Oct 30 23:46:47.433084 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:47.433052    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"]
Oct 30 23:46:47.433326 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:47.433279    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9" podUID="1ad7971e-1d0f-4404-883a-76bcf6da8b48" containerName="istio-proxy" containerID="cri-o://c19e4e5ab62393984d7997f3f0f57fe3901376375afc835a7e845529f4eb9044" gracePeriod=30
Oct 30 23:46:47.536782 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:47.536747    5078 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 23:46:47.537103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:47.537073    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c" podUID="412494b5-417a-488e-9ce4-9771d6458325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 23:46:47.709586 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:46:47.709494    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:46:48.253657 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.252494    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"]
Oct 30 23:46:48.257160 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.257134    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.310841 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.310806    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a8204d-openshift-default-dockercfg-7fmdb\""
Oct 30 23:46:48.321895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.321867    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-envoy\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.322124 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.322101    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-podinfo\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.322331 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.322313    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-credential-socket\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.322570 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.322552    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-token\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.322801 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.322774    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v52bd\" (UniqueName: \"kubernetes.io/projected/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-kube-api-access-v52bd\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.322890 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.322806    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-workload-certs\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.322890 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.322842    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-data\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.323061 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.322933    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-workload-socket\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.323061 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.322967    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istiod-ca-cert\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.371105 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.371074    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"]
Oct 30 23:46:48.445618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.437769    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-workload-socket\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.445618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.437828    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istiod-ca-cert\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.445618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.437858    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-envoy\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.445618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.437894    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-podinfo\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.445618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.437946    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-credential-socket\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.445618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.438073    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-token\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.445618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.438221    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v52bd\" (UniqueName: \"kubernetes.io/projected/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-kube-api-access-v52bd\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.445618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.438291    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-workload-certs\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.445618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.438329    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-data\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.445618 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.439480    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-workload-socket\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.453436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.453400    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istiod-ca-cert\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.454883 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.454855    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-workload-certs\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.455115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.455097    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-data\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.455488 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.455466    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-credential-socket\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.480820 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.480789    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-podinfo\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.482784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.482739    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-envoy\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.508985 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.508919    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-token\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.512170 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.512140    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v52bd\" (UniqueName: \"kubernetes.io/projected/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-kube-api-access-v52bd\") pod \"e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") " pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.555269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.555237    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:46:48.555882 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.555857    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:46:48.557439 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.557405    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:46:48.572330 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.572308    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:48.889116 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.889083    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"]
Oct 30 23:46:48.893189 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.893163    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:48.945173 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.945137    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-276bec-openshift-default-dockercfg-nkxc4\""
Oct 30 23:46:48.945560 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.945530    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-istio-data\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:48.945754 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.945579    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-workload-socket\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:48.945754 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.945722    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-credential-socket\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:48.945901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.945823    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a61efb0a-769e-4659-9452-3302535611c1-istiod-ca-cert\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:48.945901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.945887    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72gj7\" (UniqueName: \"kubernetes.io/projected/a61efb0a-769e-4659-9452-3302535611c1-kube-api-access-72gj7\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:48.946022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.945919    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-istio-envoy\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:48.946180 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.946160    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a61efb0a-769e-4659-9452-3302535611c1-istio-token\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:48.946238 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.946212    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-workload-certs\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:48.946332 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.946311    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a61efb0a-769e-4659-9452-3302535611c1-istio-podinfo\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:48.957174 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:48.957129    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"]
Oct 30 23:46:49.005949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.005924    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"]
Oct 30 23:46:49.007570 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:46:49.007544    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e7542f5_5c45_4cf1_b0b0_0f079709f0d4.slice/crio-4a7d0b5931ecc07e90fa4eb8a32d7de4271226df4fa53ce5989d5dc02c3a7c1d WatchSource:0}: Error finding container 4a7d0b5931ecc07e90fa4eb8a32d7de4271226df4fa53ce5989d5dc02c3a7c1d: Status 404 returned error can't find the container with id 4a7d0b5931ecc07e90fa4eb8a32d7de4271226df4fa53ce5989d5dc02c3a7c1d
Oct 30 23:46:49.010299 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.010270    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:49.010409 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.010333    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:49.010409 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.010363    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:49.021436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.021412    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk" event={"ID":"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4","Type":"ContainerStarted","Data":"4a7d0b5931ecc07e90fa4eb8a32d7de4271226df4fa53ce5989d5dc02c3a7c1d"}
Oct 30 23:46:49.072624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.054517    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-workload-certs\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.072624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.054585    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a61efb0a-769e-4659-9452-3302535611c1-istio-podinfo\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.072624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.054645    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-istio-data\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.072624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.054676    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-workload-socket\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.072624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.054738    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-credential-socket\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.072624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.054807    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a61efb0a-769e-4659-9452-3302535611c1-istiod-ca-cert\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.072624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.054861    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72gj7\" (UniqueName: \"kubernetes.io/projected/a61efb0a-769e-4659-9452-3302535611c1-kube-api-access-72gj7\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.072624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.054891    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-istio-envoy\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.072624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.055035    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a61efb0a-769e-4659-9452-3302535611c1-istio-token\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.072624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.055847    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-workload-certs\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.072624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.063337    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-credential-socket\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.072624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.063645    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-istio-data\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.072624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.063957    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-workload-socket\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.094010 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.093974    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-istio-envoy\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.094279 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.094247    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a61efb0a-769e-4659-9452-3302535611c1-istio-podinfo\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.094891 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.094868    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a61efb0a-769e-4659-9452-3302535611c1-istiod-ca-cert\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.132635 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.127366    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a61efb0a-769e-4659-9452-3302535611c1-istio-token\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.157684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.151933    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72gj7\" (UniqueName: \"kubernetes.io/projected/a61efb0a-769e-4659-9452-3302535611c1-kube-api-access-72gj7\") pod \"e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") " pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.220955 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.220167    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:49.662336 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.662290    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"]
Oct 30 23:46:49.667868 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.667824    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:49.668001 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.667921    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:49.668001 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:49.667992    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:46:49.725187 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:46:49.725159    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61efb0a_769e_4659_9452_3302535611c1.slice/crio-7c30fd86464c469f0459151ddab1ff781b4e8f12d15a68a641635c68eaf34e9e.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 23:46:50.031176 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:50.031080    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2" event={"ID":"a61efb0a-769e-4659-9452-3302535611c1","Type":"ContainerStarted","Data":"7c30fd86464c469f0459151ddab1ff781b4e8f12d15a68a641635c68eaf34e9e"}
Oct 30 23:46:50.031176 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:50.031142    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2" event={"ID":"a61efb0a-769e-4659-9452-3302535611c1","Type":"ContainerStarted","Data":"96e20c5d7530d763ccd745d82990c64b0e86645a2e83d17fae9a1a5b4b122f36"}
Oct 30 23:46:50.032849 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:50.032817    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk" event={"ID":"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4","Type":"ContainerStarted","Data":"eb96f3c549197e968533ae14f3252a52b9ecbae016de3e8e1818b3faef46ea93"}
Oct 30 23:46:50.097454 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:50.097418    5078 patch_prober.go:29] interesting pod/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:46:50.097649 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:50.097496    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9" podUID="1ad7971e-1d0f-4404-883a-76bcf6da8b48" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:46:50.148494 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:50.148407    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2" podStartSLOduration=2.148391272 podStartE2EDuration="2.148391272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:50.14670671 +0000 UTC m=+2396.034613063" watchObservedRunningTime="2026-10-30 23:46:50.148391272 +0000 UTC m=+2396.036297624"
Oct 30 23:46:50.223210 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:50.223173    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:50.229085 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:50.229057    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:50.271951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:50.271886    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk" podStartSLOduration=2.271866091 podStartE2EDuration="2.271866091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:46:50.269202075 +0000 UTC m=+2396.157108469" watchObservedRunningTime="2026-10-30 23:46:50.271866091 +0000 UTC m=+2396.159772445"
Oct 30 23:46:50.573053 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:50.573020    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:50.578738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:50.578711    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:51.039786 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:51.039695    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:51.039786 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:51.039756    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:51.040677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:51.040657    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:46:51.040790 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:51.040708    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:46:52.717950 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.717924    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:46:52.862559 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.862525    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-podinfo\") pod \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") "
Oct 30 23:46:52.862726 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.862566    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-workload-certs\") pod \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") "
Oct 30 23:46:52.862829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.862762    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-credential-socket\") pod \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") "
Oct 30 23:46:52.862829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.862787    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istiod-ca-cert\") pod \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") "
Oct 30 23:46:52.862829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.862816    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ad7971e-1d0f-4404-883a-76bcf6da8b48-kube-api-access-86r5d\" (UniqueName: \"kubernetes.io/projected/1ad7971e-1d0f-4404-883a-76bcf6da8b48-kube-api-access-86r5d\") pod \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") "
Oct 30 23:46:52.862999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.862834    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-data\") pod \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") "
Oct 30 23:46:52.862999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.862919    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-envoy\") pod \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") "
Oct 30 23:46:52.862999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.862920    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-workload-certs" pod "1ad7971e-1d0f-4404-883a-76bcf6da8b48" (UID: "1ad7971e-1d0f-4404-883a-76bcf6da8b48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:52.862999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.862985    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-workload-socket\") pod \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") "
Oct 30 23:46:52.863214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.863017    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-token\" (UniqueName: \"kubernetes.io/projected/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-token\") pod \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\" (UID: \"1ad7971e-1d0f-4404-883a-76bcf6da8b48\") "
Oct 30 23:46:52.863214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.863066    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-credential-socket" pod "1ad7971e-1d0f-4404-883a-76bcf6da8b48" (UID: "1ad7971e-1d0f-4404-883a-76bcf6da8b48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:52.863214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.863184    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-data" pod "1ad7971e-1d0f-4404-883a-76bcf6da8b48" (UID: "1ad7971e-1d0f-4404-883a-76bcf6da8b48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:52.863214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.863184    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istiod-ca-cert" pod "1ad7971e-1d0f-4404-883a-76bcf6da8b48" (UID: "1ad7971e-1d0f-4404-883a-76bcf6da8b48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:46:52.863563 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.863434    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-workload-socket" pod "1ad7971e-1d0f-4404-883a-76bcf6da8b48" (UID: "1ad7971e-1d0f-4404-883a-76bcf6da8b48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:52.863722 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.863674    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:52.863722 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.863696    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:52.863722 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.863705    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:52.863722 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.863718    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:52.865145 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.865125    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad7971e-1d0f-4404-883a-76bcf6da8b48-kube-api-access-86r5d" pod "1ad7971e-1d0f-4404-883a-76bcf6da8b48" (UID: "1ad7971e-1d0f-4404-883a-76bcf6da8b48"). InnerVolumeSpecName "kube-api-access-86r5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:46:52.865266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.865253    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-token" pod "1ad7971e-1d0f-4404-883a-76bcf6da8b48" (UID: "1ad7971e-1d0f-4404-883a-76bcf6da8b48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:46:52.865538 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.865519    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-podinfo" pod "1ad7971e-1d0f-4404-883a-76bcf6da8b48" (UID: "1ad7971e-1d0f-4404-883a-76bcf6da8b48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:46:52.865538 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.865525    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-envoy" pod "1ad7971e-1d0f-4404-883a-76bcf6da8b48" (UID: "1ad7971e-1d0f-4404-883a-76bcf6da8b48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:46:52.965072 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.964995    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:52.965072 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.965026    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ad7971e-1d0f-4404-883a-76bcf6da8b48-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:52.965072 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.965038    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:52.965072 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.965047    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ad7971e-1d0f-4404-883a-76bcf6da8b48-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:52.965072 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:52.965056    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86r5d\" (UniqueName: \"kubernetes.io/projected/1ad7971e-1d0f-4404-883a-76bcf6da8b48-kube-api-access-86r5d\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:46:53.114909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:53.114874    5078 generic.go:356] "Generic (PLEG): container finished" podID="1ad7971e-1d0f-4404-883a-76bcf6da8b48" containerID="c19e4e5ab62393984d7997f3f0f57fe3901376375afc835a7e845529f4eb9044" exitCode=0
Oct 30 23:46:53.115083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:53.114942    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"
Oct 30 23:46:53.115083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:53.114966    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9" event={"ID":"1ad7971e-1d0f-4404-883a-76bcf6da8b48","Type":"ContainerDied","Data":"c19e4e5ab62393984d7997f3f0f57fe3901376375afc835a7e845529f4eb9044"}
Oct 30 23:46:53.115083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:53.115059    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9" event={"ID":"1ad7971e-1d0f-4404-883a-76bcf6da8b48","Type":"ContainerDied","Data":"791c0e39bae0b9db94e346576426b232bfd4eea367f821f13b7925aa246a84ca"}
Oct 30 23:46:53.115221 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:53.115088    5078 scope.go:122] "RemoveContainer" containerID="c19e4e5ab62393984d7997f3f0f57fe3901376375afc835a7e845529f4eb9044"
Oct 30 23:46:53.126757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:53.126736    5078 scope.go:122] "RemoveContainer" containerID="380ae4650e11749b1d329d95cffe581517f0a75900bfdb25ddb8feb200711d92"
Oct 30 23:46:53.136392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:53.136360    5078 scope.go:122] "RemoveContainer" containerID="c19e4e5ab62393984d7997f3f0f57fe3901376375afc835a7e845529f4eb9044"
Oct 30 23:46:53.136665 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:46:53.136643    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c19e4e5ab62393984d7997f3f0f57fe3901376375afc835a7e845529f4eb9044\": container with ID starting with c19e4e5ab62393984d7997f3f0f57fe3901376375afc835a7e845529f4eb9044 not found: ID does not exist" containerID="c19e4e5ab62393984d7997f3f0f57fe3901376375afc835a7e845529f4eb9044"
Oct 30 23:46:53.136720 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:53.136675    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c19e4e5ab62393984d7997f3f0f57fe3901376375afc835a7e845529f4eb9044"} err="failed to get container status \"c19e4e5ab62393984d7997f3f0f57fe3901376375afc835a7e845529f4eb9044\": rpc error: code = NotFound desc = could not find container \"c19e4e5ab62393984d7997f3f0f57fe3901376375afc835a7e845529f4eb9044\": container with ID starting with c19e4e5ab62393984d7997f3f0f57fe3901376375afc835a7e845529f4eb9044 not found: ID does not exist"
Oct 30 23:46:53.136720 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:53.136692    5078 scope.go:122] "RemoveContainer" containerID="380ae4650e11749b1d329d95cffe581517f0a75900bfdb25ddb8feb200711d92"
Oct 30 23:46:53.136931 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:46:53.136918    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380ae4650e11749b1d329d95cffe581517f0a75900bfdb25ddb8feb200711d92\": container with ID starting with 380ae4650e11749b1d329d95cffe581517f0a75900bfdb25ddb8feb200711d92 not found: ID does not exist" containerID="380ae4650e11749b1d329d95cffe581517f0a75900bfdb25ddb8feb200711d92"
Oct 30 23:46:53.136977 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:53.136935    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380ae4650e11749b1d329d95cffe581517f0a75900bfdb25ddb8feb200711d92"} err="failed to get container status \"380ae4650e11749b1d329d95cffe581517f0a75900bfdb25ddb8feb200711d92\": rpc error: code = NotFound desc = could not find container \"380ae4650e11749b1d329d95cffe581517f0a75900bfdb25ddb8feb200711d92\": container with ID starting with 380ae4650e11749b1d329d95cffe581517f0a75900bfdb25ddb8feb200711d92 not found: ID does not exist"
Oct 30 23:46:53.294702 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:53.294672    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"]
Oct 30 23:46:53.303311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:53.303277    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-26ce6b-openshift-default-69f8d67887-7bfp9"]
Oct 30 23:46:54.299783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:54.299755    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ad7971e-1d0f-4404-883a-76bcf6da8b48" path="/var/lib/kubelet/pods/1ad7971e-1d0f-4404-883a-76bcf6da8b48/volumes"
Oct 30 23:46:57.006387 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:57.006360    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv"
Oct 30 23:46:58.016904 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:58.016876    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm"
Oct 30 23:46:59.029004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:59.028975    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd"]
Oct 30 23:46:59.029495 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:59.029482    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1ad7971e-1d0f-4404-883a-76bcf6da8b48" containerName="istio-proxy"
Oct 30 23:46:59.029554 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:59.029497    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ad7971e-1d0f-4404-883a-76bcf6da8b48" containerName="istio-proxy"
Oct 30 23:46:59.029626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:59.029621    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ad7971e-1d0f-4404-883a-76bcf6da8b48" containerName="istio-proxy"
Oct 30 23:46:59.034526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:59.034508    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd"
Oct 30 23:46:59.088056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:59.088019    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-276bec\""
Oct 30 23:46:59.091213 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:59.091178    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd"]
Oct 30 23:46:59.249466 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:59.249347    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-276bec-dockercfg-f97bn\""
Oct 30 23:46:59.270448 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:59.270414    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d955624a-a1d8-47f0-88d4-e1e334b21b88-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd\" (UID: \"d955624a-a1d8-47f0-88d4-e1e334b21b88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd"
Oct 30 23:46:59.270662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:59.270637    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s64bg\" (UniqueName: \"kubernetes.io/projected/d955624a-a1d8-47f0-88d4-e1e334b21b88-kube-api-access-s64bg\") pod \"payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd\" (UID: \"d955624a-a1d8-47f0-88d4-e1e334b21b88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd"
Oct 30 23:46:59.371725 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:59.371674    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d955624a-a1d8-47f0-88d4-e1e334b21b88-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd\" (UID: \"d955624a-a1d8-47f0-88d4-e1e334b21b88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd"
Oct 30 23:46:59.371923 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:59.371741    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s64bg\" (UniqueName: \"kubernetes.io/projected/d955624a-a1d8-47f0-88d4-e1e334b21b88-kube-api-access-s64bg\") pod \"payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd\" (UID: \"d955624a-a1d8-47f0-88d4-e1e334b21b88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd"
Oct 30 23:46:59.372428 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:59.372403    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d955624a-a1d8-47f0-88d4-e1e334b21b88-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd\" (UID: \"d955624a-a1d8-47f0-88d4-e1e334b21b88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd"
Oct 30 23:46:59.428415 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:59.428388    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s64bg\" (UniqueName: \"kubernetes.io/projected/d955624a-a1d8-47f0-88d4-e1e334b21b88-kube-api-access-s64bg\") pod \"payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd\" (UID: \"d955624a-a1d8-47f0-88d4-e1e334b21b88\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd"
Oct 30 23:46:59.647493 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:46:59.647410    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd"
Oct 30 23:47:00.022673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:00.022641    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd"]
Oct 30 23:47:00.025041 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:47:00.024202    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd955624a_a1d8_47f0_88d4_e1e334b21b88.slice/crio-28c302b36706e0f91b344b3493eb33b6c11da672256e6a80204df27406991bdf WatchSource:0}: Error finding container 28c302b36706e0f91b344b3493eb33b6c11da672256e6a80204df27406991bdf: Status 404 returned error can't find the container with id 28c302b36706e0f91b344b3493eb33b6c11da672256e6a80204df27406991bdf
Oct 30 23:47:00.287172 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:00.287087    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd" event={"ID":"d955624a-a1d8-47f0-88d4-e1e334b21b88","Type":"ContainerStarted","Data":"0a4b1da15b650e0014ab7fe68379066baf41cf5db2f542900e7e541a0175648e"}
Oct 30 23:47:00.287172 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:00.287140    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd" event={"ID":"d955624a-a1d8-47f0-88d4-e1e334b21b88","Type":"ContainerStarted","Data":"28c302b36706e0f91b344b3493eb33b6c11da672256e6a80204df27406991bdf"}
Oct 30 23:47:00.287172 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:00.287165    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd"
Oct 30 23:47:00.401801 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:00.401749    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd" podStartSLOduration=2.401734607 podStartE2EDuration="2.401734607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:46:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:47:00.399862492 +0000 UTC m=+2406.287768858" watchObservedRunningTime="2026-10-30 23:47:00.401734607 +0000 UTC m=+2406.289640960"
Oct 30 23:47:01.563267 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.563236    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6"]
Oct 30 23:47:01.564287 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.564267    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1ad7971e-1d0f-4404-883a-76bcf6da8b48" containerName="istio-proxy"
Oct 30 23:47:01.564413 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.564403    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ad7971e-1d0f-4404-883a-76bcf6da8b48" containerName="istio-proxy"
Oct 30 23:47:01.564676 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.564664    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ad7971e-1d0f-4404-883a-76bcf6da8b48" containerName="istio-proxy"
Oct 30 23:47:01.568189 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.568167    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6"
Oct 30 23:47:01.610641 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.610578    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhzq4\" (UniqueName: \"kubernetes.io/projected/79f476c9-ba1a-4abc-8a5e-413b6e1f1e05-kube-api-access-qhzq4\") pod \"payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6\" (UID: \"79f476c9-ba1a-4abc-8a5e-413b6e1f1e05\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6"
Oct 30 23:47:01.610809 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.610650    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f476c9-ba1a-4abc-8a5e-413b6e1f1e05-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6\" (UID: \"79f476c9-ba1a-4abc-8a5e-413b6e1f1e05\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6"
Oct 30 23:47:01.640926 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.640867    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6"]
Oct 30 23:47:01.677947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.677909    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-a8204d\""
Oct 30 23:47:01.678204 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.678184    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-a8204d-dockercfg-tc7kj\""
Oct 30 23:47:01.753478 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.750013    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhzq4\" (UniqueName: \"kubernetes.io/projected/79f476c9-ba1a-4abc-8a5e-413b6e1f1e05-kube-api-access-qhzq4\") pod \"payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6\" (UID: \"79f476c9-ba1a-4abc-8a5e-413b6e1f1e05\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6"
Oct 30 23:47:01.770478 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.770445    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f476c9-ba1a-4abc-8a5e-413b6e1f1e05-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6\" (UID: \"79f476c9-ba1a-4abc-8a5e-413b6e1f1e05\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6"
Oct 30 23:47:01.786314 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.786231    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f476c9-ba1a-4abc-8a5e-413b6e1f1e05-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6\" (UID: \"79f476c9-ba1a-4abc-8a5e-413b6e1f1e05\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6"
Oct 30 23:47:01.802103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.802050    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh"]
Oct 30 23:47:01.813178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.813152    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh"]
Oct 30 23:47:01.813415 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.813263    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh"
Oct 30 23:47:01.815418 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.815386    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhzq4\" (UniqueName: \"kubernetes.io/projected/79f476c9-ba1a-4abc-8a5e-413b6e1f1e05-kube-api-access-qhzq4\") pod \"payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6\" (UID: \"79f476c9-ba1a-4abc-8a5e-413b6e1f1e05\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6"
Oct 30 23:47:01.872060 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.872028    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9d7m\" (UniqueName: \"kubernetes.io/projected/b2da8d7e-5648-494f-8f49-e60fb6060a94-kube-api-access-p9d7m\") pod \"payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh\" (UID: \"b2da8d7e-5648-494f-8f49-e60fb6060a94\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh"
Oct 30 23:47:01.872269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.872087    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2da8d7e-5648-494f-8f49-e60fb6060a94-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh\" (UID: \"b2da8d7e-5648-494f-8f49-e60fb6060a94\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh"
Oct 30 23:47:01.881751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.881724    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6"
Oct 30 23:47:01.973721 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.973686    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9d7m\" (UniqueName: \"kubernetes.io/projected/b2da8d7e-5648-494f-8f49-e60fb6060a94-kube-api-access-p9d7m\") pod \"payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh\" (UID: \"b2da8d7e-5648-494f-8f49-e60fb6060a94\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh"
Oct 30 23:47:01.973891 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.973801    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2da8d7e-5648-494f-8f49-e60fb6060a94-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh\" (UID: \"b2da8d7e-5648-494f-8f49-e60fb6060a94\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh"
Oct 30 23:47:01.974732 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:01.974706    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2da8d7e-5648-494f-8f49-e60fb6060a94-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh\" (UID: \"b2da8d7e-5648-494f-8f49-e60fb6060a94\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh"
Oct 30 23:47:02.030527 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:02.030500    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9d7m\" (UniqueName: \"kubernetes.io/projected/b2da8d7e-5648-494f-8f49-e60fb6060a94-kube-api-access-p9d7m\") pod \"payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh\" (UID: \"b2da8d7e-5648-494f-8f49-e60fb6060a94\") " pod="openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh"
Oct 30 23:47:02.138812 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:02.138738    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh"
Oct 30 23:47:02.258816 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:02.258785    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6"]
Oct 30 23:47:02.261362 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:47:02.261327    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79f476c9_ba1a_4abc_8a5e_413b6e1f1e05.slice/crio-8f2d556cc2a9a9f1af0cb8c92108c8a71e650b857ef7a216dd8f135fddd56762 WatchSource:0}: Error finding container 8f2d556cc2a9a9f1af0cb8c92108c8a71e650b857ef7a216dd8f135fddd56762: Status 404 returned error can't find the container with id 8f2d556cc2a9a9f1af0cb8c92108c8a71e650b857ef7a216dd8f135fddd56762
Oct 30 23:47:02.343754 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:02.343726    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6" event={"ID":"79f476c9-ba1a-4abc-8a5e-413b6e1f1e05","Type":"ContainerStarted","Data":"f62ba1d2067dc090f436e98738d155e4195c77213c9f4aea781947e4edf8cec5"}
Oct 30 23:47:02.343892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:02.343794    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6"
Oct 30 23:47:02.343892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:02.343818    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6" event={"ID":"79f476c9-ba1a-4abc-8a5e-413b6e1f1e05","Type":"ContainerStarted","Data":"8f2d556cc2a9a9f1af0cb8c92108c8a71e650b857ef7a216dd8f135fddd56762"}
Oct 30 23:47:02.457380 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:02.457267    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6" podStartSLOduration=1.45724824 podStartE2EDuration="1.45724824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:47:02.457142656 +0000 UTC m=+2408.345049008" watchObservedRunningTime="2026-10-30 23:47:02.45724824 +0000 UTC m=+2408.345154593"
Oct 30 23:47:02.515865 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:02.515840    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh"]
Oct 30 23:47:02.516358 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:47:02.516338    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2da8d7e_5648_494f_8f49_e60fb6060a94.slice/crio-45e8c2980baae980bcde40b1481dc826c782d840815afe79091c636db659dd42 WatchSource:0}: Error finding container 45e8c2980baae980bcde40b1481dc826c782d840815afe79091c636db659dd42: Status 404 returned error can't find the container with id 45e8c2980baae980bcde40b1481dc826c782d840815afe79091c636db659dd42
Oct 30 23:47:03.280739 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:03.280709    5078 scope.go:122] "RemoveContainer" containerID="e6f46ca9d662f8d58e031ff5740b06d47bb3cd6dfcc931a270d04b12b39a4465"
Oct 30 23:47:03.290490 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:03.290467    5078 scope.go:122] "RemoveContainer" containerID="40a3ed4598a409cbf1498564b0adc1676d7d86fe9eec3314e37b49427fb81af0"
Oct 30 23:47:03.350728 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:03.350695    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh" event={"ID":"b2da8d7e-5648-494f-8f49-e60fb6060a94","Type":"ContainerStarted","Data":"2b05bb531d9b4f17b152769013cd14e85279496fef2f4dc525fe4630e5f6a7ee"}
Oct 30 23:47:03.350856 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:03.350750    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh" event={"ID":"b2da8d7e-5648-494f-8f49-e60fb6060a94","Type":"ContainerStarted","Data":"45e8c2980baae980bcde40b1481dc826c782d840815afe79091c636db659dd42"}
Oct 30 23:47:03.466558 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:03.466499    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh" podStartSLOduration=2.466483825 podStartE2EDuration="2.466483825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:47:03.464506906 +0000 UTC m=+2409.352413259" watchObservedRunningTime="2026-10-30 23:47:03.466483825 +0000 UTC m=+2409.354390223"
Oct 30 23:47:04.357129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:04.357098    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh"
Oct 30 23:47:11.339224 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:11.339193    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd"
Oct 30 23:47:13.353735 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:13.353711    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6"
Oct 30 23:47:15.366810 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:15.366784    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh"
Oct 30 23:47:17.591564 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:47:17.591522    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:47:22.156516 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:22.156485    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq"]
Oct 30 23:47:22.160584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:22.160568    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq"
Oct 30 23:47:22.217344 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:22.217316    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq"]
Oct 30 23:47:22.262988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:22.262956    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:47:22.262988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:22.262967    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:47:22.263185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:22.263033    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:47:22.263185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:22.263102    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:47:22.354218 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:22.354189    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g6st\" (UniqueName: \"kubernetes.io/projected/178c6907-4429-4f22-98c7-99986c942bd8-kube-api-access-9g6st\") pod \"maas-api-revoke-sub-c4e96f80-tqgpq\" (UID: \"178c6907-4429-4f22-98c7-99986c942bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq"
Oct 30 23:47:22.354386 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:22.354250    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178c6907-4429-4f22-98c7-99986c942bd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4e96f80-tqgpq\" (UID: \"178c6907-4429-4f22-98c7-99986c942bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq"
Oct 30 23:47:22.354445 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:22.354423    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178c6907-4429-4f22-98c7-99986c942bd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4e96f80-tqgpq\" (UID: \"178c6907-4429-4f22-98c7-99986c942bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq"
Oct 30 23:47:22.456182 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:22.456100    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178c6907-4429-4f22-98c7-99986c942bd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4e96f80-tqgpq\" (UID: \"178c6907-4429-4f22-98c7-99986c942bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq"
Oct 30 23:47:22.456344 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:22.456259    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178c6907-4429-4f22-98c7-99986c942bd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4e96f80-tqgpq\" (UID: \"178c6907-4429-4f22-98c7-99986c942bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq"
Oct 30 23:47:22.456392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:22.456377    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g6st\" (UniqueName: \"kubernetes.io/projected/178c6907-4429-4f22-98c7-99986c942bd8-kube-api-access-9g6st\") pod \"maas-api-revoke-sub-c4e96f80-tqgpq\" (UID: \"178c6907-4429-4f22-98c7-99986c942bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq"
Oct 30 23:47:22.457005 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:22.456983    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178c6907-4429-4f22-98c7-99986c942bd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4e96f80-tqgpq\" (UID: \"178c6907-4429-4f22-98c7-99986c942bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq"
Oct 30 23:47:22.458634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:22.458590    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178c6907-4429-4f22-98c7-99986c942bd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4e96f80-tqgpq\" (UID: \"178c6907-4429-4f22-98c7-99986c942bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq"
Oct 30 23:47:22.513817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:22.513795    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g6st\" (UniqueName: \"kubernetes.io/projected/178c6907-4429-4f22-98c7-99986c942bd8-kube-api-access-9g6st\") pod \"maas-api-revoke-sub-c4e96f80-tqgpq\" (UID: \"178c6907-4429-4f22-98c7-99986c942bd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq"
Oct 30 23:47:22.774537 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:22.774459    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq"
Oct 30 23:47:23.150183 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:23.150155    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq"]
Oct 30 23:47:23.151176 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:47:23.151138    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178c6907_4429_4f22_98c7_99986c942bd8.slice/crio-c3b10d8bc218811b8bfcfa9ca148a0bfc0cb782c7eef2ebb8796fc30c468360a WatchSource:0}: Error finding container c3b10d8bc218811b8bfcfa9ca148a0bfc0cb782c7eef2ebb8796fc30c468360a: Status 404 returned error can't find the container with id c3b10d8bc218811b8bfcfa9ca148a0bfc0cb782c7eef2ebb8796fc30c468360a
Oct 30 23:47:23.477439 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:23.477405    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq" event={"ID":"178c6907-4429-4f22-98c7-99986c942bd8","Type":"ContainerStarted","Data":"c3b10d8bc218811b8bfcfa9ca148a0bfc0cb782c7eef2ebb8796fc30c468360a"}
Oct 30 23:47:24.484949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:24.484916    5078 generic.go:356] "Generic (PLEG): container finished" podID="178c6907-4429-4f22-98c7-99986c942bd8" containerID="3d82b30e7194fdc4a9a0d2293ab14f248dc137fd1617360f5912dbf7065e463a" exitCode=0
Oct 30 23:47:24.485366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:24.485003    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq" event={"ID":"178c6907-4429-4f22-98c7-99986c942bd8","Type":"ContainerDied","Data":"3d82b30e7194fdc4a9a0d2293ab14f248dc137fd1617360f5912dbf7065e463a"}
Oct 30 23:47:25.702937 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:25.702913    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq"
Oct 30 23:47:25.791373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:25.791346    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/178c6907-4429-4f22-98c7-99986c942bd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178c6907-4429-4f22-98c7-99986c942bd8-maas-api-openshift-ca\") pod \"178c6907-4429-4f22-98c7-99986c942bd8\" (UID: \"178c6907-4429-4f22-98c7-99986c942bd8\") "
Oct 30 23:47:25.791520 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:25.791463    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/178c6907-4429-4f22-98c7-99986c942bd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178c6907-4429-4f22-98c7-99986c942bd8-maas-api-xks-ca\") pod \"178c6907-4429-4f22-98c7-99986c942bd8\" (UID: \"178c6907-4429-4f22-98c7-99986c942bd8\") "
Oct 30 23:47:25.791520 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:25.791506    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/178c6907-4429-4f22-98c7-99986c942bd8-kube-api-access-9g6st\" (UniqueName: \"kubernetes.io/projected/178c6907-4429-4f22-98c7-99986c942bd8-kube-api-access-9g6st\") pod \"178c6907-4429-4f22-98c7-99986c942bd8\" (UID: \"178c6907-4429-4f22-98c7-99986c942bd8\") "
Oct 30 23:47:25.791782 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:25.791758    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/178c6907-4429-4f22-98c7-99986c942bd8-maas-api-openshift-ca" pod "178c6907-4429-4f22-98c7-99986c942bd8" (UID: "178c6907-4429-4f22-98c7-99986c942bd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:47:25.792037 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:25.792015    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/178c6907-4429-4f22-98c7-99986c942bd8-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:47:25.793685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:25.793652    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/178c6907-4429-4f22-98c7-99986c942bd8-maas-api-xks-ca" pod "178c6907-4429-4f22-98c7-99986c942bd8" (UID: "178c6907-4429-4f22-98c7-99986c942bd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:47:25.793807 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:25.793755    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/178c6907-4429-4f22-98c7-99986c942bd8-kube-api-access-9g6st" pod "178c6907-4429-4f22-98c7-99986c942bd8" (UID: "178c6907-4429-4f22-98c7-99986c942bd8"). InnerVolumeSpecName "kube-api-access-9g6st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:47:25.893322 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:25.893288    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/178c6907-4429-4f22-98c7-99986c942bd8-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:47:25.893322 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:25.893318    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g6st\" (UniqueName: \"kubernetes.io/projected/178c6907-4429-4f22-98c7-99986c942bd8-kube-api-access-9g6st\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:47:26.503059 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:26.503031    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq"
Oct 30 23:47:26.503250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:26.503061    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq" event={"ID":"178c6907-4429-4f22-98c7-99986c942bd8","Type":"ContainerDied","Data":"c3b10d8bc218811b8bfcfa9ca148a0bfc0cb782c7eef2ebb8796fc30c468360a"}
Oct 30 23:47:26.503250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:26.503105    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3b10d8bc218811b8bfcfa9ca148a0bfc0cb782c7eef2ebb8796fc30c468360a"
Oct 30 23:47:33.385415 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.385337    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc"]
Oct 30 23:47:33.385873 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.385859    5078 cpu_manager.go:414] "containerMap: removing container" podUID="178c6907-4429-4f22-98c7-99986c942bd8" containerName="revoke-keys"
Oct 30 23:47:33.385930 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.385874    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="178c6907-4429-4f22-98c7-99986c942bd8" containerName="revoke-keys"
Oct 30 23:47:33.385968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.385956    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="178c6907-4429-4f22-98c7-99986c942bd8" containerName="revoke-keys"
Oct 30 23:47:33.389387 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.389372    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc"
Oct 30 23:47:33.445650 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.445614    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc"]
Oct 30 23:47:33.484352 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.484321    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnxrd\" (UniqueName: \"kubernetes.io/projected/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-kube-api-access-rnxrd\") pod \"maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc\" (UID: \"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc"
Oct 30 23:47:33.484628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.484583    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc\" (UID: \"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc"
Oct 30 23:47:33.484871 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.484846    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc\" (UID: \"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc"
Oct 30 23:47:33.491350 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.491330    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:47:33.491449 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.491344    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:47:33.491449 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.491329    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:47:33.491449 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.491354    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:47:33.586028 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.586003    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnxrd\" (UniqueName: \"kubernetes.io/projected/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-kube-api-access-rnxrd\") pod \"maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc\" (UID: \"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc"
Oct 30 23:47:33.586200 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.586136    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc\" (UID: \"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc"
Oct 30 23:47:33.586258 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.586193    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc\" (UID: \"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc"
Oct 30 23:47:33.586871 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.586845    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc\" (UID: \"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc"
Oct 30 23:47:33.588642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.588619    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc\" (UID: \"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc"
Oct 30 23:47:33.642444 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.642389    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnxrd\" (UniqueName: \"kubernetes.io/projected/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-kube-api-access-rnxrd\") pod \"maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc\" (UID: \"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc"
Oct 30 23:47:33.703520 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:33.703494    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc"
Oct 30 23:47:34.078446 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:34.078421    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc"]
Oct 30 23:47:34.079629 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:47:34.079590    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0fd94f_6aef_4c8d_9b79_4d29cb6437cd.slice/crio-126e8535ada88497b9064e56ee23dc3bf7c414ad730ba2fd95d348f71b4a23e3 WatchSource:0}: Error finding container 126e8535ada88497b9064e56ee23dc3bf7c414ad730ba2fd95d348f71b4a23e3: Status 404 returned error can't find the container with id 126e8535ada88497b9064e56ee23dc3bf7c414ad730ba2fd95d348f71b4a23e3
Oct 30 23:47:34.557928 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:34.557892    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc" event={"ID":"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd","Type":"ContainerStarted","Data":"7785750cb5febc980da72f883b469889b091aacefa26f131de05725b29ed4a9a"}
Oct 30 23:47:34.558317 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:34.557958    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc" event={"ID":"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd","Type":"ContainerStarted","Data":"126e8535ada88497b9064e56ee23dc3bf7c414ad730ba2fd95d348f71b4a23e3"}
Oct 30 23:47:34.672200 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:34.672105    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc" podStartSLOduration=1.672089599 podStartE2EDuration="1.672089599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:47:34.671242596 +0000 UTC m=+2440.559148953" watchObservedRunningTime="2026-10-30 23:47:34.672089599 +0000 UTC m=+2440.559995953"
Oct 30 23:47:35.566872 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:35.566842    5078 generic.go:356] "Generic (PLEG): container finished" podID="cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd" containerID="7785750cb5febc980da72f883b469889b091aacefa26f131de05725b29ed4a9a" exitCode=0
Oct 30 23:47:35.567287 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:35.566926    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc" event={"ID":"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd","Type":"ContainerDied","Data":"7785750cb5febc980da72f883b469889b091aacefa26f131de05725b29ed4a9a"}
Oct 30 23:47:36.728888 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:36.728861    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc"
Oct 30 23:47:36.829225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:36.829132    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-maas-api-openshift-ca\") pod \"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd\" (UID: \"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd\") "
Oct 30 23:47:36.829396 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:36.829295    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-maas-api-xks-ca\") pod \"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd\" (UID: \"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd\") "
Oct 30 23:47:36.829396 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:36.829373    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-kube-api-access-rnxrd\" (UniqueName: \"kubernetes.io/projected/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-kube-api-access-rnxrd\") pod \"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd\" (UID: \"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd\") "
Oct 30 23:47:36.829539 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:36.829492    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-maas-api-openshift-ca" pod "cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd" (UID: "cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:47:36.829798 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:36.829777    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:47:36.831644 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:36.831612    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-maas-api-xks-ca" pod "cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd" (UID: "cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:47:36.831751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:36.831673    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-kube-api-access-rnxrd" pod "cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd" (UID: "cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd"). InnerVolumeSpecName "kube-api-access-rnxrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:47:36.930670 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:36.930640    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnxrd\" (UniqueName: \"kubernetes.io/projected/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-kube-api-access-rnxrd\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:47:36.930670 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:36.930667    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:47:37.581192 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:37.581156    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc"
Oct 30 23:47:37.581192 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:37.581179    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc" event={"ID":"cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd","Type":"ContainerDied","Data":"126e8535ada88497b9064e56ee23dc3bf7c414ad730ba2fd95d348f71b4a23e3"}
Oct 30 23:47:37.581418 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:37.581223    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="126e8535ada88497b9064e56ee23dc3bf7c414ad730ba2fd95d348f71b4a23e3"
Oct 30 23:47:37.831919 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:37.831831    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"]
Oct 30 23:47:37.845794 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:37.845699    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"]
Oct 30 23:47:43.913912 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:43.913871    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd"]
Oct 30 23:47:47.593689 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:47:47.593652    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:47:48.914564 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:48.914507    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd" podUID="7b8a41ae-c270-4b4a-b758-f4dee7dd748f" containerName="payload-processing" containerID="cri-o://7b20a1eff5a6932950059bce529e7af552520a4853ad74f31ab76f5175cd8515" gracePeriod=25
Oct 30 23:47:52.854975 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:52.854931    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main" containerID="cri-o://4e8e4b3009db4c7fb58d883885f100f1dc0d62b59cdb00d64926e7cadbdee971" gracePeriod=45
Oct 30 23:47:52.865898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:52.865848    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main" containerID="cri-o://9f0d3978956c04892b30eaddcf0119d11a904cc295b34dddd3a44532e6bf7457" gracePeriod=45
Oct 30 23:47:52.918112 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:52.918073    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": dial tcp 10.133.0.44:8000: connect: connection refused"
Oct 30 23:47:53.140925 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.140888    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:47:53.170178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.170145    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:47:53.208638 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.208581    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-model-cache\") pod \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") "
Oct 30 23:47:53.208840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.208714    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-tmp-dir\") pod \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") "
Oct 30 23:47:53.208920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.208830    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-model-cache" pod "f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" (UID: "f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:47:53.208920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.208883    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-kube-api-access-lmhsl\" (UniqueName: \"kubernetes.io/projected/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-kube-api-access-lmhsl\") pod \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") "
Oct 30 23:47:53.209032 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.208954    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-tmp-dir" pod "f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" (UID: "f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:47:53.209032 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.208977    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-home\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-home\") pod \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") "
Oct 30 23:47:53.209032 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.209023    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-tls-certs\") pod \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") "
Oct 30 23:47:53.209180 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.209104    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-dshm\") pod \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\" (UID: \"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67\") "
Oct 30 23:47:53.209180 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.209172    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-home" pod "f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" (UID: "f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:47:53.209753 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.209693    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:47:53.209753 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.209719    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:47:53.209753 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.209733    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:47:53.211270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.211248    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-tls-certs" pod "f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" (UID: "f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:47:53.211364 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.211327    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-kube-api-access-lmhsl" pod "f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" (UID: "f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67"). InnerVolumeSpecName "kube-api-access-lmhsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:47:53.211474 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.211458    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-dshm" pod "f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" (UID: "f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:47:53.310457 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.310423    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-dshm\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-dshm\") pod \"731a9563-7dbd-47a6-80b5-4a7d48433786\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") "
Oct 30 23:47:53.310657 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.310541    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-tmp-dir\") pod \"731a9563-7dbd-47a6-80b5-4a7d48433786\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") "
Oct 30 23:47:53.310657 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.310651    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-home\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-home\") pod \"731a9563-7dbd-47a6-80b5-4a7d48433786\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") "
Oct 30 23:47:53.310799 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.310736    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-model-cache\") pod \"731a9563-7dbd-47a6-80b5-4a7d48433786\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") "
Oct 30 23:47:53.310886 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.310855    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-tmp-dir" pod "731a9563-7dbd-47a6-80b5-4a7d48433786" (UID: "731a9563-7dbd-47a6-80b5-4a7d48433786"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:47:53.310886 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.310862    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731a9563-7dbd-47a6-80b5-4a7d48433786-kube-api-access-ffd6t\" (UniqueName: \"kubernetes.io/projected/731a9563-7dbd-47a6-80b5-4a7d48433786-kube-api-access-ffd6t\") pod \"731a9563-7dbd-47a6-80b5-4a7d48433786\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") "
Oct 30 23:47:53.311086 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.310948    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/731a9563-7dbd-47a6-80b5-4a7d48433786-tls-certs\" (UniqueName: \"kubernetes.io/secret/731a9563-7dbd-47a6-80b5-4a7d48433786-tls-certs\") pod \"731a9563-7dbd-47a6-80b5-4a7d48433786\" (UID: \"731a9563-7dbd-47a6-80b5-4a7d48433786\") "
Oct 30 23:47:53.311151 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.311105    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-home" pod "731a9563-7dbd-47a6-80b5-4a7d48433786" (UID: "731a9563-7dbd-47a6-80b5-4a7d48433786"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:47:53.311212 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.311152    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-model-cache" pod "731a9563-7dbd-47a6-80b5-4a7d48433786" (UID: "731a9563-7dbd-47a6-80b5-4a7d48433786"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:47:53.311693 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.311672    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:47:53.311838 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.311697    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:47:53.311838 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.311713    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmhsl\" (UniqueName: \"kubernetes.io/projected/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-kube-api-access-lmhsl\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:47:53.311838 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.311727    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:47:53.311838 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.311740    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:47:53.311838 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.311752    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:47:53.313381 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.313359    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-dshm" pod "731a9563-7dbd-47a6-80b5-4a7d48433786" (UID: "731a9563-7dbd-47a6-80b5-4a7d48433786"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:47:53.313584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.313563    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/731a9563-7dbd-47a6-80b5-4a7d48433786-tls-certs" pod "731a9563-7dbd-47a6-80b5-4a7d48433786" (UID: "731a9563-7dbd-47a6-80b5-4a7d48433786"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:47:53.313674 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.313632    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731a9563-7dbd-47a6-80b5-4a7d48433786-kube-api-access-ffd6t" pod "731a9563-7dbd-47a6-80b5-4a7d48433786" (UID: "731a9563-7dbd-47a6-80b5-4a7d48433786"). InnerVolumeSpecName "kube-api-access-ffd6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:47:53.420447 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.420359    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffd6t\" (UniqueName: \"kubernetes.io/projected/731a9563-7dbd-47a6-80b5-4a7d48433786-kube-api-access-ffd6t\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:47:53.420447 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.420401    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/731a9563-7dbd-47a6-80b5-4a7d48433786-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:47:53.420447 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.420417    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/731a9563-7dbd-47a6-80b5-4a7d48433786-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:47:53.773318 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.773227    5078 generic.go:356] "Generic (PLEG): container finished" podID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerID="9f0d3978956c04892b30eaddcf0119d11a904cc295b34dddd3a44532e6bf7457" exitCode=0
Oct 30 23:47:53.773318 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.773288    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"
Oct 30 23:47:53.773318 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.773310    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" event={"ID":"731a9563-7dbd-47a6-80b5-4a7d48433786","Type":"ContainerDied","Data":"9f0d3978956c04892b30eaddcf0119d11a904cc295b34dddd3a44532e6bf7457"}
Oct 30 23:47:53.773643 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.773350    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl" event={"ID":"731a9563-7dbd-47a6-80b5-4a7d48433786","Type":"ContainerDied","Data":"21c8bb0cf34fee04e75b729c921c0c3c333d8b83c66be429d5daa0c0964e7503"}
Oct 30 23:47:53.773643 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.773368    5078 scope.go:122] "RemoveContainer" containerID="9f0d3978956c04892b30eaddcf0119d11a904cc295b34dddd3a44532e6bf7457"
Oct 30 23:47:53.774642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.774616    5078 generic.go:356] "Generic (PLEG): container finished" podID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerID="4e8e4b3009db4c7fb58d883885f100f1dc0d62b59cdb00d64926e7cadbdee971" exitCode=0
Oct 30 23:47:53.774751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.774644    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" event={"ID":"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67","Type":"ContainerDied","Data":"4e8e4b3009db4c7fb58d883885f100f1dc0d62b59cdb00d64926e7cadbdee971"}
Oct 30 23:47:53.774751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.774679    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7" event={"ID":"f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67","Type":"ContainerDied","Data":"fab36fffe3befd0f37da622613f358c6aa8a78bdbd387508bc6be987a33f58ea"}
Oct 30 23:47:53.774751 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.774696    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"
Oct 30 23:47:53.784312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.784291    5078 scope.go:122] "RemoveContainer" containerID="9f0d3978956c04892b30eaddcf0119d11a904cc295b34dddd3a44532e6bf7457"
Oct 30 23:47:53.784718 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:47:53.784677    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0d3978956c04892b30eaddcf0119d11a904cc295b34dddd3a44532e6bf7457\": container with ID starting with 9f0d3978956c04892b30eaddcf0119d11a904cc295b34dddd3a44532e6bf7457 not found: ID does not exist" containerID="9f0d3978956c04892b30eaddcf0119d11a904cc295b34dddd3a44532e6bf7457"
Oct 30 23:47:53.784803 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.784725    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0d3978956c04892b30eaddcf0119d11a904cc295b34dddd3a44532e6bf7457"} err="failed to get container status \"9f0d3978956c04892b30eaddcf0119d11a904cc295b34dddd3a44532e6bf7457\": rpc error: code = NotFound desc = could not find container \"9f0d3978956c04892b30eaddcf0119d11a904cc295b34dddd3a44532e6bf7457\": container with ID starting with 9f0d3978956c04892b30eaddcf0119d11a904cc295b34dddd3a44532e6bf7457 not found: ID does not exist"
Oct 30 23:47:53.784803 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.784744    5078 scope.go:122] "RemoveContainer" containerID="4e8e4b3009db4c7fb58d883885f100f1dc0d62b59cdb00d64926e7cadbdee971"
Oct 30 23:47:53.793881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.793863    5078 scope.go:122] "RemoveContainer" containerID="4e8e4b3009db4c7fb58d883885f100f1dc0d62b59cdb00d64926e7cadbdee971"
Oct 30 23:47:53.794225 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:47:53.794202    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e8e4b3009db4c7fb58d883885f100f1dc0d62b59cdb00d64926e7cadbdee971\": container with ID starting with 4e8e4b3009db4c7fb58d883885f100f1dc0d62b59cdb00d64926e7cadbdee971 not found: ID does not exist" containerID="4e8e4b3009db4c7fb58d883885f100f1dc0d62b59cdb00d64926e7cadbdee971"
Oct 30 23:47:53.794279 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.794235    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e8e4b3009db4c7fb58d883885f100f1dc0d62b59cdb00d64926e7cadbdee971"} err="failed to get container status \"4e8e4b3009db4c7fb58d883885f100f1dc0d62b59cdb00d64926e7cadbdee971\": rpc error: code = NotFound desc = could not find container \"4e8e4b3009db4c7fb58d883885f100f1dc0d62b59cdb00d64926e7cadbdee971\": container with ID starting with 4e8e4b3009db4c7fb58d883885f100f1dc0d62b59cdb00d64926e7cadbdee971 not found: ID does not exist"
Oct 30 23:47:53.901402 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.901369    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd" podUID="7b8a41ae-c270-4b4a-b758-f4dee7dd748f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:47:53.938296 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.938260    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"]
Oct 30 23:47:53.941620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:53.941584    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3c9613/premium-w2-3c9613-kserve-7844659cf5-wknt7"]
Oct 30 23:47:54.049683 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:54.049652    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"]
Oct 30 23:47:54.056020 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:54.055994    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-3c9613/simulator-w2-3c9613-kserve-56f659f86b-fgwbl"]
Oct 30 23:47:54.300086 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:54.300013    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" path="/var/lib/kubelet/pods/731a9563-7dbd-47a6-80b5-4a7d48433786/volumes"
Oct 30 23:47:54.300433 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:47:54.300419    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" path="/var/lib/kubelet/pods/f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67/volumes"
Oct 30 23:48:03.901131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:03.901092    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd" podUID="7b8a41ae-c270-4b4a-b758-f4dee7dd748f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:48:06.031158 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.031121    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"]
Oct 30 23:48:06.031724 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.031709    5078 cpu_manager.go:414] "containerMap: removing container" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main"
Oct 30 23:48:06.031784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.031726    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main"
Oct 30 23:48:06.031784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.031737    5078 cpu_manager.go:414] "containerMap: removing container" podUID="cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd" containerName="revoke-keys"
Oct 30 23:48:06.031784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.031742    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd" containerName="revoke-keys"
Oct 30 23:48:06.031784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.031751    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main"
Oct 30 23:48:06.031784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.031757    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main"
Oct 30 23:48:06.031975 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.031852    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="731a9563-7dbd-47a6-80b5-4a7d48433786" containerName="main"
Oct 30 23:48:06.031975 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.031861    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd" containerName="revoke-keys"
Oct 30 23:48:06.031975 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.031869    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d4add7-ab33-4dbd-b9aa-b238a9c3fb67" containerName="main"
Oct 30 23:48:06.036713 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.036695    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.117319 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.117292    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.117458 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.117327    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7x47\" (UniqueName: \"kubernetes.io/projected/4e81b860-0772-48e5-a0a2-8e89095eb245-kube-api-access-f7x47\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.117458 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.117440    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.117550 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.117491    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.117588 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.117541    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e81b860-0772-48e5-a0a2-8e89095eb245-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.117665 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.117652    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.117718 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.117707    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.117758 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.117739    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.117803 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.117760    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.135029 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.135004    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"]
Oct 30 23:48:06.218970 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.218941    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.219157 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.218979    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.219157 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.219109    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.219157 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.219144    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7x47\" (UniqueName: \"kubernetes.io/projected/4e81b860-0772-48e5-a0a2-8e89095eb245-kube-api-access-f7x47\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.219354 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.219229    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.219354 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.219284    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.219354 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.219315    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e81b860-0772-48e5-a0a2-8e89095eb245-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.219537 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.219378    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.219537 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.219400    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.219537 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.219489    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.219733 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.219665    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.219798 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.219750    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.219798 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.219788    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.220159 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.220137    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e81b860-0772-48e5-a0a2-8e89095eb245-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.222097 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.222071    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.222227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.222071    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.275990 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.275957    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.276303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.276285    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7x47\" (UniqueName: \"kubernetes.io/projected/4e81b860-0772-48e5-a0a2-8e89095eb245-kube-api-access-f7x47\") pod \"maas-default-gateway-openshift-default-5df48fcc99-n6bxx\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.349867 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.349823    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:06.784716 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.784685    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"]
Oct 30 23:48:06.785701 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:48:06.785672    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e81b860_0772_48e5_a0a2_8e89095eb245.slice/crio-22c13ae39fae1405df501e67122d22bfa5864856c5eafd3dfffb0867bb099483 WatchSource:0}: Error finding container 22c13ae39fae1405df501e67122d22bfa5864856c5eafd3dfffb0867bb099483: Status 404 returned error can't find the container with id 22c13ae39fae1405df501e67122d22bfa5864856c5eafd3dfffb0867bb099483
Oct 30 23:48:06.788393 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.788363    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:48:06.788466 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.788428    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:48:06.788466 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.788456    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:48:06.868665 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.868634    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx" event={"ID":"4e81b860-0772-48e5-a0a2-8e89095eb245","Type":"ContainerStarted","Data":"d8637f4c9112212ff8665bc9f8c21f8b885c699afedd95192efd031429c2d743"}
Oct 30 23:48:06.868837 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.868687    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx" event={"ID":"4e81b860-0772-48e5-a0a2-8e89095eb245","Type":"ContainerStarted","Data":"22c13ae39fae1405df501e67122d22bfa5864856c5eafd3dfffb0867bb099483"}
Oct 30 23:48:06.987963 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:06.987904    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx" podStartSLOduration=1.987882467 podStartE2EDuration="1.987882467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:06.985377446 +0000 UTC m=+2472.873283794" watchObservedRunningTime="2026-10-30 23:48:06.987882467 +0000 UTC m=+2472.875788825"
Oct 30 23:48:07.355409 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:07.355367    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:07.356249 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:07.356223    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n6bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 23:48:07.356408 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:07.356386    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx" podUID="4e81b860-0772-48e5-a0a2-8e89095eb245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 23:48:08.351145 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:08.351111    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n6bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 23:48:08.351310 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:08.351162    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx" podUID="4e81b860-0772-48e5-a0a2-8e89095eb245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 23:48:09.218667 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.218638    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd"
Oct 30 23:48:09.307988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.307953    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8a41ae-c270-4b4a-b758-f4dee7dd748f-kube-api-access-lbjkg\" (UniqueName: \"kubernetes.io/projected/7b8a41ae-c270-4b4a-b758-f4dee7dd748f-kube-api-access-lbjkg\") pod \"7b8a41ae-c270-4b4a-b758-f4dee7dd748f\" (UID: \"7b8a41ae-c270-4b4a-b758-f4dee7dd748f\") "
Oct 30 23:48:09.308168 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.308127    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b8a41ae-c270-4b4a-b758-f4dee7dd748f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8a41ae-c270-4b4a-b758-f4dee7dd748f-plugins-config-volume\") pod \"7b8a41ae-c270-4b4a-b758-f4dee7dd748f\" (UID: \"7b8a41ae-c270-4b4a-b758-f4dee7dd748f\") "
Oct 30 23:48:09.308444 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.308421    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8a41ae-c270-4b4a-b758-f4dee7dd748f-plugins-config-volume" pod "7b8a41ae-c270-4b4a-b758-f4dee7dd748f" (UID: "7b8a41ae-c270-4b4a-b758-f4dee7dd748f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:48:09.308646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.308627    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b8a41ae-c270-4b4a-b758-f4dee7dd748f-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:48:09.310316 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.310185    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8a41ae-c270-4b4a-b758-f4dee7dd748f-kube-api-access-lbjkg" pod "7b8a41ae-c270-4b4a-b758-f4dee7dd748f" (UID: "7b8a41ae-c270-4b4a-b758-f4dee7dd748f"). InnerVolumeSpecName "kube-api-access-lbjkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:48:09.350829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.350800    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n6bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 23:48:09.351013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.350870    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx" podUID="4e81b860-0772-48e5-a0a2-8e89095eb245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 23:48:09.409890 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.409861    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbjkg\" (UniqueName: \"kubernetes.io/projected/7b8a41ae-c270-4b4a-b758-f4dee7dd748f-kube-api-access-lbjkg\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:48:09.733646 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.733264    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb"]
Oct 30 23:48:09.734128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.734104    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7b8a41ae-c270-4b4a-b758-f4dee7dd748f" containerName="payload-processing"
Oct 30 23:48:09.734216 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.734131    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b8a41ae-c270-4b4a-b758-f4dee7dd748f" containerName="payload-processing"
Oct 30 23:48:09.734327 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.734314    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b8a41ae-c270-4b4a-b758-f4dee7dd748f" containerName="payload-processing"
Oct 30 23:48:09.737898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.737881    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb"
Oct 30 23:48:09.795038 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.795001    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:48:09.802880 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.802835    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:48:09.803768 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.803740    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:48:09.803913 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.803853    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:48:09.815449 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.813662    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb"]
Oct 30 23:48:09.819812 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.819777    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c51425db-16c3-47d4-99f8-1ae1b831772f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14c236eb-dxtbb\" (UID: \"c51425db-16c3-47d4-99f8-1ae1b831772f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb"
Oct 30 23:48:09.820142 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.820119    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m"]
Oct 30 23:48:09.820341 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.820163    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6zjj\" (UniqueName: \"kubernetes.io/projected/c51425db-16c3-47d4-99f8-1ae1b831772f-kube-api-access-x6zjj\") pod \"maas-api-revoke-sub-14c236eb-dxtbb\" (UID: \"c51425db-16c3-47d4-99f8-1ae1b831772f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb"
Oct 30 23:48:09.820462 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.820410    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c51425db-16c3-47d4-99f8-1ae1b831772f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14c236eb-dxtbb\" (UID: \"c51425db-16c3-47d4-99f8-1ae1b831772f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb"
Oct 30 23:48:09.824195 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.824178    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m"
Oct 30 23:48:09.921247 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.921217    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967439c8-ed26-4000-bacd-809cc04fe909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd79610e-g929m\" (UID: \"967439c8-ed26-4000-bacd-809cc04fe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m"
Oct 30 23:48:09.921442 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.921431    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967439c8-ed26-4000-bacd-809cc04fe909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd79610e-g929m\" (UID: \"967439c8-ed26-4000-bacd-809cc04fe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m"
Oct 30 23:48:09.921522 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.921514    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c51425db-16c3-47d4-99f8-1ae1b831772f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14c236eb-dxtbb\" (UID: \"c51425db-16c3-47d4-99f8-1ae1b831772f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb"
Oct 30 23:48:09.921648 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.921633    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86cn5\" (UniqueName: \"kubernetes.io/projected/967439c8-ed26-4000-bacd-809cc04fe909-kube-api-access-86cn5\") pod \"maas-api-revoke-sub-cd79610e-g929m\" (UID: \"967439c8-ed26-4000-bacd-809cc04fe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m"
Oct 30 23:48:09.921792 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.921781    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6zjj\" (UniqueName: \"kubernetes.io/projected/c51425db-16c3-47d4-99f8-1ae1b831772f-kube-api-access-x6zjj\") pod \"maas-api-revoke-sub-14c236eb-dxtbb\" (UID: \"c51425db-16c3-47d4-99f8-1ae1b831772f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb"
Oct 30 23:48:09.921890 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.921880    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c51425db-16c3-47d4-99f8-1ae1b831772f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14c236eb-dxtbb\" (UID: \"c51425db-16c3-47d4-99f8-1ae1b831772f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb"
Oct 30 23:48:09.923895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.923866    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m"]
Oct 30 23:48:09.924451 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.924416    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c51425db-16c3-47d4-99f8-1ae1b831772f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14c236eb-dxtbb\" (UID: \"c51425db-16c3-47d4-99f8-1ae1b831772f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb"
Oct 30 23:48:09.924983 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.924959    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c51425db-16c3-47d4-99f8-1ae1b831772f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14c236eb-dxtbb\" (UID: \"c51425db-16c3-47d4-99f8-1ae1b831772f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb"
Oct 30 23:48:09.926129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.926100    5078 generic.go:356] "Generic (PLEG): container finished" podID="7b8a41ae-c270-4b4a-b758-f4dee7dd748f" containerID="7b20a1eff5a6932950059bce529e7af552520a4853ad74f31ab76f5175cd8515" exitCode=0
Oct 30 23:48:09.926241 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.926196    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd"
Oct 30 23:48:09.926241 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.926199    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd" event={"ID":"7b8a41ae-c270-4b4a-b758-f4dee7dd748f","Type":"ContainerDied","Data":"7b20a1eff5a6932950059bce529e7af552520a4853ad74f31ab76f5175cd8515"}
Oct 30 23:48:09.926368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.926248    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd" event={"ID":"7b8a41ae-c270-4b4a-b758-f4dee7dd748f","Type":"ContainerDied","Data":"7e80752e131f522be5eef24424e61a5365685ec935a340736dd8668bfcc9c867"}
Oct 30 23:48:09.926368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.926278    5078 scope.go:122] "RemoveContainer" containerID="7b20a1eff5a6932950059bce529e7af552520a4853ad74f31ab76f5175cd8515"
Oct 30 23:48:09.942831 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.942800    5078 scope.go:122] "RemoveContainer" containerID="7b20a1eff5a6932950059bce529e7af552520a4853ad74f31ab76f5175cd8515"
Oct 30 23:48:09.943102 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:48:09.943085    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b20a1eff5a6932950059bce529e7af552520a4853ad74f31ab76f5175cd8515\": container with ID starting with 7b20a1eff5a6932950059bce529e7af552520a4853ad74f31ab76f5175cd8515 not found: ID does not exist" containerID="7b20a1eff5a6932950059bce529e7af552520a4853ad74f31ab76f5175cd8515"
Oct 30 23:48:09.943182 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:09.943107    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b20a1eff5a6932950059bce529e7af552520a4853ad74f31ab76f5175cd8515"} err="failed to get container status \"7b20a1eff5a6932950059bce529e7af552520a4853ad74f31ab76f5175cd8515\": rpc error: code = NotFound desc = could not find container \"7b20a1eff5a6932950059bce529e7af552520a4853ad74f31ab76f5175cd8515\": container with ID starting with 7b20a1eff5a6932950059bce529e7af552520a4853ad74f31ab76f5175cd8515 not found: ID does not exist"
Oct 30 23:48:10.010551 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:10.010476    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6zjj\" (UniqueName: \"kubernetes.io/projected/c51425db-16c3-47d4-99f8-1ae1b831772f-kube-api-access-x6zjj\") pod \"maas-api-revoke-sub-14c236eb-dxtbb\" (UID: \"c51425db-16c3-47d4-99f8-1ae1b831772f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb"
Oct 30 23:48:10.022932 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:10.022905    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967439c8-ed26-4000-bacd-809cc04fe909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd79610e-g929m\" (UID: \"967439c8-ed26-4000-bacd-809cc04fe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m"
Oct 30 23:48:10.023062 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:10.022944    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967439c8-ed26-4000-bacd-809cc04fe909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd79610e-g929m\" (UID: \"967439c8-ed26-4000-bacd-809cc04fe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m"
Oct 30 23:48:10.023105 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:10.023074    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86cn5\" (UniqueName: \"kubernetes.io/projected/967439c8-ed26-4000-bacd-809cc04fe909-kube-api-access-86cn5\") pod \"maas-api-revoke-sub-cd79610e-g929m\" (UID: \"967439c8-ed26-4000-bacd-809cc04fe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m"
Oct 30 23:48:10.023636 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:10.023616    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967439c8-ed26-4000-bacd-809cc04fe909-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd79610e-g929m\" (UID: \"967439c8-ed26-4000-bacd-809cc04fe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m"
Oct 30 23:48:10.026771 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:10.026754    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967439c8-ed26-4000-bacd-809cc04fe909-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd79610e-g929m\" (UID: \"967439c8-ed26-4000-bacd-809cc04fe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m"
Oct 30 23:48:10.051042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:10.051016    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb"
Oct 30 23:48:10.093633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:10.092908    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86cn5\" (UniqueName: \"kubernetes.io/projected/967439c8-ed26-4000-bacd-809cc04fe909-kube-api-access-86cn5\") pod \"maas-api-revoke-sub-cd79610e-g929m\" (UID: \"967439c8-ed26-4000-bacd-809cc04fe909\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m"
Oct 30 23:48:10.119247 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:10.118576    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd"]
Oct 30 23:48:10.122463 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:10.122431    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-3c9613-5f5d5b5c7-jjdvd"]
Oct 30 23:48:10.139536 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:10.139500    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m"
Oct 30 23:48:10.354996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:10.351015    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n6bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 23:48:10.354996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:10.351069    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx" podUID="4e81b860-0772-48e5-a0a2-8e89095eb245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 23:48:10.373624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:10.356196    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b8a41ae-c270-4b4a-b758-f4dee7dd748f" path="/var/lib/kubelet/pods/7b8a41ae-c270-4b4a-b758-f4dee7dd748f/volumes"
Oct 30 23:48:10.839029 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:10.838985    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb"]
Oct 30 23:48:10.946399 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:48:10.946371    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod967439c8_ed26_4000_bacd_809cc04fe909.slice/crio-59aff097a07cfb8b30b9564b7df6d9b4e63f3d86b01300d50cf2c420098c290f WatchSource:0}: Error finding container 59aff097a07cfb8b30b9564b7df6d9b4e63f3d86b01300d50cf2c420098c290f: Status 404 returned error can't find the container with id 59aff097a07cfb8b30b9564b7df6d9b4e63f3d86b01300d50cf2c420098c290f
Oct 30 23:48:10.948509 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:10.948486    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m"]
Oct 30 23:48:10.950789 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:10.950766    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb" event={"ID":"c51425db-16c3-47d4-99f8-1ae1b831772f","Type":"ContainerStarted","Data":"82c2bae279c4822e5a604c7a013d3126838644a2580516e4d86c36191655253a"}
Oct 30 23:48:11.351176 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:11.351095    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n6bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 23:48:11.351176 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:11.351150    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx" podUID="4e81b860-0772-48e5-a0a2-8e89095eb245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 23:48:12.045368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:12.045325    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb" event={"ID":"c51425db-16c3-47d4-99f8-1ae1b831772f","Type":"ContainerStarted","Data":"415f60846e23484e5c9735aca92f31df1164e95c4df0c232f168f510bd78f568"}
Oct 30 23:48:12.047087 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:12.047068    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m" event={"ID":"967439c8-ed26-4000-bacd-809cc04fe909","Type":"ContainerStarted","Data":"42bc6256b70b7bb98570e0a973f171533441320fc5d9752897d5c2a4e50915b8"}
Oct 30 23:48:12.047153 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:12.047107    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m" event={"ID":"967439c8-ed26-4000-bacd-809cc04fe909","Type":"ContainerStarted","Data":"59aff097a07cfb8b30b9564b7df6d9b4e63f3d86b01300d50cf2c420098c290f"}
Oct 30 23:48:12.243250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:12.243179    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb" podStartSLOduration=3.243153664 podStartE2EDuration="3.243153664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:12.239681451 +0000 UTC m=+2478.127587804" watchObservedRunningTime="2026-10-30 23:48:12.243153664 +0000 UTC m=+2478.131060029"
Oct 30 23:48:12.350677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:12.350646    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n6bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 23:48:12.350807 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:12.350709    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx" podUID="4e81b860-0772-48e5-a0a2-8e89095eb245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 23:48:13.054953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:13.054909    5078 generic.go:356] "Generic (PLEG): container finished" podID="c51425db-16c3-47d4-99f8-1ae1b831772f" containerID="415f60846e23484e5c9735aca92f31df1164e95c4df0c232f168f510bd78f568" exitCode=0
Oct 30 23:48:13.055391 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:13.055003    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb" event={"ID":"c51425db-16c3-47d4-99f8-1ae1b831772f","Type":"ContainerDied","Data":"415f60846e23484e5c9735aca92f31df1164e95c4df0c232f168f510bd78f568"}
Oct 30 23:48:13.062626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:13.056659    5078 generic.go:356] "Generic (PLEG): container finished" podID="967439c8-ed26-4000-bacd-809cc04fe909" containerID="42bc6256b70b7bb98570e0a973f171533441320fc5d9752897d5c2a4e50915b8" exitCode=0
Oct 30 23:48:13.062928 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:13.060769    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m" event={"ID":"967439c8-ed26-4000-bacd-809cc04fe909","Type":"ContainerDied","Data":"42bc6256b70b7bb98570e0a973f171533441320fc5d9752897d5c2a4e50915b8"}
Oct 30 23:48:13.179622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:13.178715    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m" podStartSLOduration=4.178694604 podStartE2EDuration="4.178694604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:48:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:48:12.385826653 +0000 UTC m=+2478.273733009" watchObservedRunningTime="2026-10-30 23:48:13.178694604 +0000 UTC m=+2479.066600957"
Oct 30 23:48:13.350981 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:13.350892    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n6bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 23:48:13.350981 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:13.350939    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx" podUID="4e81b860-0772-48e5-a0a2-8e89095eb245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 23:48:14.351263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:14.351179    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n6bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused" start-of-body=
Oct 30 23:48:14.352115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:14.352084    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx" podUID="4e81b860-0772-48e5-a0a2-8e89095eb245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": dial tcp 10.133.0.223:15021: connect: connection refused"
Oct 30 23:48:14.722673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:14.720799    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb"
Oct 30 23:48:14.852811 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:14.852784    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m"
Oct 30 23:48:14.927056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:14.918024    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c51425db-16c3-47d4-99f8-1ae1b831772f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c51425db-16c3-47d4-99f8-1ae1b831772f-maas-api-xks-ca\") pod \"c51425db-16c3-47d4-99f8-1ae1b831772f\" (UID: \"c51425db-16c3-47d4-99f8-1ae1b831772f\") "
Oct 30 23:48:14.927056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:14.918081    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c51425db-16c3-47d4-99f8-1ae1b831772f-kube-api-access-x6zjj\" (UniqueName: \"kubernetes.io/projected/c51425db-16c3-47d4-99f8-1ae1b831772f-kube-api-access-x6zjj\") pod \"c51425db-16c3-47d4-99f8-1ae1b831772f\" (UID: \"c51425db-16c3-47d4-99f8-1ae1b831772f\") "
Oct 30 23:48:14.927056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:14.918251    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c51425db-16c3-47d4-99f8-1ae1b831772f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c51425db-16c3-47d4-99f8-1ae1b831772f-maas-api-openshift-ca\") pod \"c51425db-16c3-47d4-99f8-1ae1b831772f\" (UID: \"c51425db-16c3-47d4-99f8-1ae1b831772f\") "
Oct 30 23:48:14.927056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:14.918279    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/967439c8-ed26-4000-bacd-809cc04fe909-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967439c8-ed26-4000-bacd-809cc04fe909-maas-api-openshift-ca\") pod \"967439c8-ed26-4000-bacd-809cc04fe909\" (UID: \"967439c8-ed26-4000-bacd-809cc04fe909\") "
Oct 30 23:48:14.927056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:14.918310    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/967439c8-ed26-4000-bacd-809cc04fe909-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967439c8-ed26-4000-bacd-809cc04fe909-maas-api-xks-ca\") pod \"967439c8-ed26-4000-bacd-809cc04fe909\" (UID: \"967439c8-ed26-4000-bacd-809cc04fe909\") "
Oct 30 23:48:14.927056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:14.918407    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/967439c8-ed26-4000-bacd-809cc04fe909-kube-api-access-86cn5\" (UniqueName: \"kubernetes.io/projected/967439c8-ed26-4000-bacd-809cc04fe909-kube-api-access-86cn5\") pod \"967439c8-ed26-4000-bacd-809cc04fe909\" (UID: \"967439c8-ed26-4000-bacd-809cc04fe909\") "
Oct 30 23:48:14.931852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:14.931576    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c51425db-16c3-47d4-99f8-1ae1b831772f-maas-api-openshift-ca" pod "c51425db-16c3-47d4-99f8-1ae1b831772f" (UID: "c51425db-16c3-47d4-99f8-1ae1b831772f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:48:14.936323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:14.936296    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/967439c8-ed26-4000-bacd-809cc04fe909-kube-api-access-86cn5" pod "967439c8-ed26-4000-bacd-809cc04fe909" (UID: "967439c8-ed26-4000-bacd-809cc04fe909"). InnerVolumeSpecName "kube-api-access-86cn5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:48:14.936455 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:14.936398    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c51425db-16c3-47d4-99f8-1ae1b831772f-maas-api-xks-ca" pod "c51425db-16c3-47d4-99f8-1ae1b831772f" (UID: "c51425db-16c3-47d4-99f8-1ae1b831772f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:48:14.936588 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:14.936566    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/967439c8-ed26-4000-bacd-809cc04fe909-maas-api-openshift-ca" pod "967439c8-ed26-4000-bacd-809cc04fe909" (UID: "967439c8-ed26-4000-bacd-809cc04fe909"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:48:14.937013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:14.936990    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c51425db-16c3-47d4-99f8-1ae1b831772f-kube-api-access-x6zjj" pod "c51425db-16c3-47d4-99f8-1ae1b831772f" (UID: "c51425db-16c3-47d4-99f8-1ae1b831772f"). InnerVolumeSpecName "kube-api-access-x6zjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:48:14.938019 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:14.937992    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/967439c8-ed26-4000-bacd-809cc04fe909-maas-api-xks-ca" pod "967439c8-ed26-4000-bacd-809cc04fe909" (UID: "967439c8-ed26-4000-bacd-809cc04fe909"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:48:15.023666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:15.020573    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c51425db-16c3-47d4-99f8-1ae1b831772f-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:48:15.023666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:15.020631    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/967439c8-ed26-4000-bacd-809cc04fe909-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:48:15.023666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:15.020649    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/967439c8-ed26-4000-bacd-809cc04fe909-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:48:15.023666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:15.020665    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86cn5\" (UniqueName: \"kubernetes.io/projected/967439c8-ed26-4000-bacd-809cc04fe909-kube-api-access-86cn5\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:48:15.023666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:15.020679    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c51425db-16c3-47d4-99f8-1ae1b831772f-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:48:15.023666 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:15.020694    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6zjj\" (UniqueName: \"kubernetes.io/projected/c51425db-16c3-47d4-99f8-1ae1b831772f-kube-api-access-x6zjj\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:48:15.121684 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:15.121656    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m"
Oct 30 23:48:15.122103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:15.121677    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m" event={"ID":"967439c8-ed26-4000-bacd-809cc04fe909","Type":"ContainerDied","Data":"59aff097a07cfb8b30b9564b7df6d9b4e63f3d86b01300d50cf2c420098c290f"}
Oct 30 23:48:15.122250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:15.122133    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59aff097a07cfb8b30b9564b7df6d9b4e63f3d86b01300d50cf2c420098c290f"
Oct 30 23:48:15.129630 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:15.125034    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb" event={"ID":"c51425db-16c3-47d4-99f8-1ae1b831772f","Type":"ContainerDied","Data":"82c2bae279c4822e5a604c7a013d3126838644a2580516e4d86c36191655253a"}
Oct 30 23:48:15.129630 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:15.125082    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82c2bae279c4822e5a604c7a013d3126838644a2580516e4d86c36191655253a"
Oct 30 23:48:15.129630 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:15.125142    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb"
Oct 30 23:48:16.394718 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:16.394680    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-n6bxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:48:16.395226 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:16.394747    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx" podUID="4e81b860-0772-48e5-a0a2-8e89095eb245" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:48:16.522630 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:16.521113    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:16.522630 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:16.522491    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:16.523471 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:16.523448    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:48:17.601331 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:48:17.601287    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:48:19.785273 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:19.785195    5078 patch_prober.go:29] interesting pod/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 23:48:19.785273 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:19.785243    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz" podUID="fc1e8c37-0f6c-4a6e-b877-43a235878ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 23:48:20.170145 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:20.170113    5078 generic.go:356] "Generic (PLEG): container finished" podID="fc1e8c37-0f6c-4a6e-b877-43a235878ca8" containerID="e76df04832f9ecc5ef738d9f9ce5aaee769e833ab8f4ea1f09235962e2ab58e0" exitCode=0
Oct 30 23:48:20.170374 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:20.170197    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz" event={"ID":"fc1e8c37-0f6c-4a6e-b877-43a235878ca8","Type":"ContainerDied","Data":"e76df04832f9ecc5ef738d9f9ce5aaee769e833ab8f4ea1f09235962e2ab58e0"}
Oct 30 23:48:20.170711 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:20.170687    5078 scope.go:122] "RemoveContainer" containerID="e76df04832f9ecc5ef738d9f9ce5aaee769e833ab8f4ea1f09235962e2ab58e0"
Oct 30 23:48:20.171993 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:20.171974    5078 generic.go:356] "Generic (PLEG): container finished" podID="d6d43311-ddb3-4d7f-901d-025555bc9e37" containerID="afb312b3d4f255bee0b333827314ef9ab4f8f0002a021337ca8cd2b0360f6e1a" exitCode=0
Oct 30 23:48:20.172109 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:20.172017    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm" event={"ID":"d6d43311-ddb3-4d7f-901d-025555bc9e37","Type":"ContainerDied","Data":"afb312b3d4f255bee0b333827314ef9ab4f8f0002a021337ca8cd2b0360f6e1a"}
Oct 30 23:48:20.172323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:20.172296    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:48:20.172397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:20.172363    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:48:20.172461 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:20.172392    5078 scope.go:122] "RemoveContainer" containerID="afb312b3d4f255bee0b333827314ef9ab4f8f0002a021337ca8cd2b0360f6e1a"
Oct 30 23:48:20.172517 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:20.172395    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:48:20.173850 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:20.173829    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:48:20.173920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:20.173886    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:48:20.173961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:20.173919    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:48:21.199902 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:21.199863    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz" event={"ID":"fc1e8c37-0f6c-4a6e-b877-43a235878ca8","Type":"ContainerStarted","Data":"e8773c79bd0076eb3559abf644ef5e2d49f1a6dc48404bd8a171769d59276a0f"}
Oct 30 23:48:21.202998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:21.202969    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm" event={"ID":"d6d43311-ddb3-4d7f-901d-025555bc9e37","Type":"ContainerStarted","Data":"1e852f569dfb880230e338689edd66ec274aace33ed62b0bb7d6e9e8b42dd211"}
Oct 30 23:48:21.685620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:21.684797    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:48:21.688757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:21.688721    5078 patch_prober.go:29] interesting pod/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 23:48:21.689924 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:21.689891    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm" podUID="d6d43311-ddb3-4d7f-901d-025555bc9e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 23:48:22.081045 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:22.081015    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:48:22.081424 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:22.081399    5078 patch_prober.go:29] interesting pod/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 23:48:22.081650 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:22.081622    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz" podUID="fc1e8c37-0f6c-4a6e-b877-43a235878ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 23:48:22.680113 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:22.680073    5078 patch_prober.go:29] interesting pod/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 23:48:22.680809 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:22.680778    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm" podUID="d6d43311-ddb3-4d7f-901d-025555bc9e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 23:48:23.080631 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:23.078005    5078 patch_prober.go:29] interesting pod/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 23:48:23.080631 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:23.078067    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz" podUID="fc1e8c37-0f6c-4a6e-b877-43a235878ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 23:48:23.677048 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:23.677007    5078 patch_prober.go:29] interesting pod/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 23:48:23.677382 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:23.677347    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm" podUID="d6d43311-ddb3-4d7f-901d-025555bc9e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 23:48:24.080622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:24.077004    5078 patch_prober.go:29] interesting pod/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 23:48:24.080622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:24.077065    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz" podUID="fc1e8c37-0f6c-4a6e-b877-43a235878ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 23:48:24.680041 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:24.680002    5078 patch_prober.go:29] interesting pod/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 23:48:24.680348 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:24.680316    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm" podUID="d6d43311-ddb3-4d7f-901d-025555bc9e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 23:48:25.074993 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:25.074952    5078 patch_prober.go:29] interesting pod/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 23:48:25.075271 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:25.075240    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz" podUID="fc1e8c37-0f6c-4a6e-b877-43a235878ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 23:48:25.711663 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:25.710117    5078 patch_prober.go:29] interesting pod/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused" start-of-body=
Oct 30 23:48:25.711663 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:25.710199    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm" podUID="d6d43311-ddb3-4d7f-901d-025555bc9e37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.213:15021/healthz/ready\": dial tcp 10.133.0.213:15021: connect: connection refused"
Oct 30 23:48:26.078036 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:26.077998    5078 patch_prober.go:29] interesting pod/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body=
Oct 30 23:48:26.078337 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:26.078308    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz" podUID="fc1e8c37-0f6c-4a6e-b877-43a235878ca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused"
Oct 30 23:48:26.681643 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:26.681579    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:48:26.681924 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:26.681901    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:48:26.682653 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:26.682622    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:48:27.079340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:27.079312    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:48:27.079848 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:27.079831    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:48:27.080653 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:27.080634    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:48:37.081772 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:37.081731    5078 patch_prober.go:29] interesting pod/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:48:37.082222 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:37.081798    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk" podUID="4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:48:37.082222 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:37.081846    5078 patch_prober.go:29] interesting pod/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:48:37.082222 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:37.081875    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2" podUID="a61efb0a-769e-4659-9452-3302535611c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:48:37.547422 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:37.547390    5078 generic.go:356] "Generic (PLEG): container finished" podID="4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" containerID="eb96f3c549197e968533ae14f3252a52b9ecbae016de3e8e1818b3faef46ea93" exitCode=0
Oct 30 23:48:37.547682 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:37.547459    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk" event={"ID":"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4","Type":"ContainerDied","Data":"eb96f3c549197e968533ae14f3252a52b9ecbae016de3e8e1818b3faef46ea93"}
Oct 30 23:48:37.547986 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:37.547969    5078 scope.go:122] "RemoveContainer" containerID="eb96f3c549197e968533ae14f3252a52b9ecbae016de3e8e1818b3faef46ea93"
Oct 30 23:48:37.549633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:37.549586    5078 generic.go:356] "Generic (PLEG): container finished" podID="a61efb0a-769e-4659-9452-3302535611c1" containerID="7c30fd86464c469f0459151ddab1ff781b4e8f12d15a68a641635c68eaf34e9e" exitCode=0
Oct 30 23:48:37.549744 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:37.549646    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2" event={"ID":"a61efb0a-769e-4659-9452-3302535611c1","Type":"ContainerDied","Data":"7c30fd86464c469f0459151ddab1ff781b4e8f12d15a68a641635c68eaf34e9e"}
Oct 30 23:48:37.549819 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:37.549789    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:48:37.549894 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:37.549877    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:48:37.549949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:37.549934    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:48:37.550081 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:37.550066    5078 scope.go:122] "RemoveContainer" containerID="7c30fd86464c469f0459151ddab1ff781b4e8f12d15a68a641635c68eaf34e9e"
Oct 30 23:48:37.551423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:37.551398    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:48:37.551505 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:37.551461    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:48:37.551559 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:37.551504    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:48:38.585122 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:38.585082    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2" event={"ID":"a61efb0a-769e-4659-9452-3302535611c1","Type":"ContainerStarted","Data":"c92cd887dce7113ee13a2c54563703d7f9cdbba1fb05ff3491a731e769fde412"}
Oct 30 23:48:38.608862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:38.608824    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk" event={"ID":"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4","Type":"ContainerStarted","Data":"c3ad267bca85d209b8890fde01b83ebeed8edc4fd2d82ba3554370e09830f6f5"}
Oct 30 23:48:39.222946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:39.222908    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:48:39.223166 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:39.223144    5078 patch_prober.go:29] interesting pod/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 23:48:39.223215 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:39.223202    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2" podUID="a61efb0a-769e-4659-9452-3302535611c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 23:48:39.577626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:39.572967    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:48:39.595621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:39.582857    5078 patch_prober.go:29] interesting pod/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 23:48:39.595621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:39.582913    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk" podUID="4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 23:48:40.223367 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:40.223328    5078 patch_prober.go:29] interesting pod/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 23:48:40.223562 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:40.223378    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2" podUID="a61efb0a-769e-4659-9452-3302535611c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 23:48:40.579419 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:40.578937    5078 patch_prober.go:29] interesting pod/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 23:48:40.579419 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:40.579035    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk" podUID="4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 23:48:41.223563 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:41.223528    5078 patch_prober.go:29] interesting pod/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 23:48:41.223955 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:41.223583    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2" podUID="a61efb0a-769e-4659-9452-3302535611c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 23:48:41.581622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:41.577948    5078 patch_prober.go:29] interesting pod/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 23:48:41.581622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:41.578009    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk" podUID="4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 23:48:42.222861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:42.222826    5078 patch_prober.go:29] interesting pod/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 23:48:42.223057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:42.222879    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2" podUID="a61efb0a-769e-4659-9452-3302535611c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 23:48:42.577623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:42.575754    5078 patch_prober.go:29] interesting pod/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 23:48:42.577623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:42.575853    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk" podUID="4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 23:48:43.223403 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:43.223370    5078 patch_prober.go:29] interesting pod/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused" start-of-body=
Oct 30 23:48:43.223625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:43.223418    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2" podUID="a61efb0a-769e-4659-9452-3302535611c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.217:15021/healthz/ready\": dial tcp 10.133.0.217:15021: connect: connection refused"
Oct 30 23:48:43.574621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:43.573976    5078 patch_prober.go:29] interesting pod/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body=
Oct 30 23:48:43.574621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:43.574061    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk" podUID="4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused"
Oct 30 23:48:44.227567 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:44.227535    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:48:44.228080 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:44.228049    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:48:44.228875 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:44.228858    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:48:44.584028 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:44.583999    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:48:44.584339 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:44.584323    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:48:44.585410 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:44.585390    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:48:47.594490 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:48:47.594456    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:48:55.990046 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:55.990009    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw"]
Oct 30 23:48:55.990580 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:55.990535    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c51425db-16c3-47d4-99f8-1ae1b831772f" containerName="revoke-keys"
Oct 30 23:48:55.990580 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:55.990546    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c51425db-16c3-47d4-99f8-1ae1b831772f" containerName="revoke-keys"
Oct 30 23:48:55.990580 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:55.990556    5078 cpu_manager.go:414] "containerMap: removing container" podUID="967439c8-ed26-4000-bacd-809cc04fe909" containerName="revoke-keys"
Oct 30 23:48:55.990580 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:55.990561    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="967439c8-ed26-4000-bacd-809cc04fe909" containerName="revoke-keys"
Oct 30 23:48:55.990759 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:55.990658    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c51425db-16c3-47d4-99f8-1ae1b831772f" containerName="revoke-keys"
Oct 30 23:48:55.990759 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:55.990666    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="967439c8-ed26-4000-bacd-809cc04fe909" containerName="revoke-keys"
Oct 30 23:48:55.994074 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:55.994056    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw"
Oct 30 23:48:56.046225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:56.046183    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:48:56.096072 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:56.096043    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:48:56.096227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:56.096078    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:48:56.096227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:56.096142    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:48:56.099417 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:56.099396    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw"]
Oct 30 23:48:56.101775 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:56.101750    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2549ce-4cdf-4965-917e-957a14cd7265-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw\" (UID: \"7a2549ce-4cdf-4965-917e-957a14cd7265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw"
Oct 30 23:48:56.101888 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:56.101814    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2549ce-4cdf-4965-917e-957a14cd7265-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw\" (UID: \"7a2549ce-4cdf-4965-917e-957a14cd7265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw"
Oct 30 23:48:56.101934 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:56.101907    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p2cg\" (UniqueName: \"kubernetes.io/projected/7a2549ce-4cdf-4965-917e-957a14cd7265-kube-api-access-7p2cg\") pod \"maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw\" (UID: \"7a2549ce-4cdf-4965-917e-957a14cd7265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw"
Oct 30 23:48:56.202798 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:56.202773    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2549ce-4cdf-4965-917e-957a14cd7265-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw\" (UID: \"7a2549ce-4cdf-4965-917e-957a14cd7265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw"
Oct 30 23:48:56.202933 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:56.202814    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p2cg\" (UniqueName: \"kubernetes.io/projected/7a2549ce-4cdf-4965-917e-957a14cd7265-kube-api-access-7p2cg\") pod \"maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw\" (UID: \"7a2549ce-4cdf-4965-917e-957a14cd7265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw"
Oct 30 23:48:56.203021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:56.203002    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2549ce-4cdf-4965-917e-957a14cd7265-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw\" (UID: \"7a2549ce-4cdf-4965-917e-957a14cd7265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw"
Oct 30 23:48:56.203454 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:56.203434    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2549ce-4cdf-4965-917e-957a14cd7265-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw\" (UID: \"7a2549ce-4cdf-4965-917e-957a14cd7265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw"
Oct 30 23:48:56.205551 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:56.205524    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2549ce-4cdf-4965-917e-957a14cd7265-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw\" (UID: \"7a2549ce-4cdf-4965-917e-957a14cd7265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw"
Oct 30 23:48:56.263167 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:56.263107    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p2cg\" (UniqueName: \"kubernetes.io/projected/7a2549ce-4cdf-4965-917e-957a14cd7265-kube-api-access-7p2cg\") pod \"maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw\" (UID: \"7a2549ce-4cdf-4965-917e-957a14cd7265\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw"
Oct 30 23:48:56.307026 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:56.307007    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw"
Oct 30 23:48:56.681359 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:56.681332    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw"]
Oct 30 23:48:56.682521 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:48:56.682488    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2549ce_4cdf_4965_917e_957a14cd7265.slice/crio-d04dbc21ffd33004ec566e9e5aef6df05782d900c20ee5253505cd24de119e5c WatchSource:0}: Error finding container d04dbc21ffd33004ec566e9e5aef6df05782d900c20ee5253505cd24de119e5c: Status 404 returned error can't find the container with id d04dbc21ffd33004ec566e9e5aef6df05782d900c20ee5253505cd24de119e5c
Oct 30 23:48:56.759315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:56.759283    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw" event={"ID":"7a2549ce-4cdf-4965-917e-957a14cd7265","Type":"ContainerStarted","Data":"d04dbc21ffd33004ec566e9e5aef6df05782d900c20ee5253505cd24de119e5c"}
Oct 30 23:48:57.767487 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:57.767449    5078 generic.go:356] "Generic (PLEG): container finished" podID="7a2549ce-4cdf-4965-917e-957a14cd7265" containerID="27612c6f5f5b7dc217b7ed0c4c0397bd625a1504fe3f319286d121b5f882f4a6" exitCode=0
Oct 30 23:48:57.767924 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:57.767543    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw" event={"ID":"7a2549ce-4cdf-4965-917e-957a14cd7265","Type":"ContainerDied","Data":"27612c6f5f5b7dc217b7ed0c4c0397bd625a1504fe3f319286d121b5f882f4a6"}
Oct 30 23:48:59.076114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:59.076081    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw"
Oct 30 23:48:59.157026 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:59.156991    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a2549ce-4cdf-4965-917e-957a14cd7265-kube-api-access-7p2cg\" (UniqueName: \"kubernetes.io/projected/7a2549ce-4cdf-4965-917e-957a14cd7265-kube-api-access-7p2cg\") pod \"7a2549ce-4cdf-4965-917e-957a14cd7265\" (UID: \"7a2549ce-4cdf-4965-917e-957a14cd7265\") "
Oct 30 23:48:59.157237 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:59.157049    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a2549ce-4cdf-4965-917e-957a14cd7265-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2549ce-4cdf-4965-917e-957a14cd7265-maas-api-openshift-ca\") pod \"7a2549ce-4cdf-4965-917e-957a14cd7265\" (UID: \"7a2549ce-4cdf-4965-917e-957a14cd7265\") "
Oct 30 23:48:59.157324 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:59.157294    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a2549ce-4cdf-4965-917e-957a14cd7265-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2549ce-4cdf-4965-917e-957a14cd7265-maas-api-xks-ca\") pod \"7a2549ce-4cdf-4965-917e-957a14cd7265\" (UID: \"7a2549ce-4cdf-4965-917e-957a14cd7265\") "
Oct 30 23:48:59.157579 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:59.157549    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a2549ce-4cdf-4965-917e-957a14cd7265-maas-api-openshift-ca" pod "7a2549ce-4cdf-4965-917e-957a14cd7265" (UID: "7a2549ce-4cdf-4965-917e-957a14cd7265"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:48:59.157872 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:59.157849    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a2549ce-4cdf-4965-917e-957a14cd7265-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:48:59.159395 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:59.159365    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2549ce-4cdf-4965-917e-957a14cd7265-kube-api-access-7p2cg" pod "7a2549ce-4cdf-4965-917e-957a14cd7265" (UID: "7a2549ce-4cdf-4965-917e-957a14cd7265"). InnerVolumeSpecName "kube-api-access-7p2cg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:48:59.159495 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:59.159475    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2549ce-4cdf-4965-917e-957a14cd7265-maas-api-xks-ca" pod "7a2549ce-4cdf-4965-917e-957a14cd7265" (UID: "7a2549ce-4cdf-4965-917e-957a14cd7265"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:48:59.259236 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:59.259197    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a2549ce-4cdf-4965-917e-957a14cd7265-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:48:59.259236 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:59.259231    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p2cg\" (UniqueName: \"kubernetes.io/projected/7a2549ce-4cdf-4965-917e-957a14cd7265-kube-api-access-7p2cg\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:48:59.784251 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:59.784222    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw"
Oct 30 23:48:59.784433 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:59.784255    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw" event={"ID":"7a2549ce-4cdf-4965-917e-957a14cd7265","Type":"ContainerDied","Data":"d04dbc21ffd33004ec566e9e5aef6df05782d900c20ee5253505cd24de119e5c"}
Oct 30 23:48:59.784433 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:48:59.784301    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d04dbc21ffd33004ec566e9e5aef6df05782d900c20ee5253505cd24de119e5c"
Oct 30 23:49:03.367462 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:03.367388    5078 scope.go:122] "RemoveContainer" containerID="0e5b38f592fadbd3596440a4916de19c1943d903fbd6e934848a10e53eba81a7"
Oct 30 23:49:03.377216 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:03.377193    5078 scope.go:122] "RemoveContainer" containerID="0fecb3f2c8898997df58451c03079a25d87383410f7d2129ca96a22fb165f8b7"
Oct 30 23:49:03.386587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:03.386568    5078 scope.go:122] "RemoveContainer" containerID="b1f694f9aae6dc0d25121618008122dfca8c9b8a447fd07ad61c23de3c798aae"
Oct 30 23:49:03.395251 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:03.395223    5078 scope.go:122] "RemoveContainer" containerID="c71fdcb6205e2922f02e1e720182801068ccd928a2008cf3489899ede16bb683"
Oct 30 23:49:03.404502 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:03.404487    5078 scope.go:122] "RemoveContainer" containerID="6a48c6925313c77111ab381634a9e51394430d5be13c5151fdad24d99ec71986"
Oct 30 23:49:08.843328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:08.843297    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"]
Oct 30 23:49:08.843860 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:08.843846    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7a2549ce-4cdf-4965-917e-957a14cd7265" containerName="revoke-keys"
Oct 30 23:49:08.843911 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:08.843861    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a2549ce-4cdf-4965-917e-957a14cd7265" containerName="revoke-keys"
Oct 30 23:49:08.843962 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:08.843953    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a2549ce-4cdf-4965-917e-957a14cd7265" containerName="revoke-keys"
Oct 30 23:49:08.848899 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:08.848884    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:08.904995 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:08.904965    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"]
Oct 30 23:49:08.916719 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:08.916693    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5e3f9d\"/\"openshift-service-ca.crt\""
Oct 30 23:49:08.920770 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:08.920733    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-tmp-dir\") pod \"test-model-5e3f9d-kserve-6fd5c9f455-cmh2h\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") " pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:08.920954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:08.920783    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-tls-certs\") pod \"test-model-5e3f9d-kserve-6fd5c9f455-cmh2h\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") " pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:08.920954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:08.920925    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xcfw\" (UniqueName: \"kubernetes.io/projected/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-kube-api-access-7xcfw\") pod \"test-model-5e3f9d-kserve-6fd5c9f455-cmh2h\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") " pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:08.921063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:08.920965    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-home\") pod \"test-model-5e3f9d-kserve-6fd5c9f455-cmh2h\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") " pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:08.921063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:08.920996    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-model-cache\") pod \"test-model-5e3f9d-kserve-6fd5c9f455-cmh2h\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") " pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:08.921143 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:08.921125    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-dshm\") pod \"test-model-5e3f9d-kserve-6fd5c9f455-cmh2h\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") " pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:08.950858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:08.950822    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5e3f9d\"/\"kube-root-ca.crt\""
Oct 30 23:49:08.950858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:08.950822    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5e3f9d\"/\"default-dockercfg-bq78k\""
Oct 30 23:49:08.951013 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:08.950858    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5e3f9d\"/\"test-model-5e3f9d-kserve-self-signed-certs\""
Oct 30 23:49:09.022099 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:09.022075    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xcfw\" (UniqueName: \"kubernetes.io/projected/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-kube-api-access-7xcfw\") pod \"test-model-5e3f9d-kserve-6fd5c9f455-cmh2h\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") " pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:09.022206 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:09.022103    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-home\") pod \"test-model-5e3f9d-kserve-6fd5c9f455-cmh2h\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") " pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:09.022206 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:09.022126    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-model-cache\") pod \"test-model-5e3f9d-kserve-6fd5c9f455-cmh2h\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") " pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:09.022206 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:09.022192    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-dshm\") pod \"test-model-5e3f9d-kserve-6fd5c9f455-cmh2h\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") " pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:09.022399 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:09.022379    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-tmp-dir\") pod \"test-model-5e3f9d-kserve-6fd5c9f455-cmh2h\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") " pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:09.022452 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:09.022422    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-tls-certs\") pod \"test-model-5e3f9d-kserve-6fd5c9f455-cmh2h\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") " pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:09.022535 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:09.022503    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-home\") pod \"test-model-5e3f9d-kserve-6fd5c9f455-cmh2h\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") " pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:09.022631 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:09.022529    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-model-cache\") pod \"test-model-5e3f9d-kserve-6fd5c9f455-cmh2h\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") " pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:09.022758 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:09.022737    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-tmp-dir\") pod \"test-model-5e3f9d-kserve-6fd5c9f455-cmh2h\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") " pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:09.024541 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:09.024515    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-dshm\") pod \"test-model-5e3f9d-kserve-6fd5c9f455-cmh2h\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") " pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:09.024929 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:09.024907    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-tls-certs\") pod \"test-model-5e3f9d-kserve-6fd5c9f455-cmh2h\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") " pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:09.079529 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:09.079498    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xcfw\" (UniqueName: \"kubernetes.io/projected/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-kube-api-access-7xcfw\") pod \"test-model-5e3f9d-kserve-6fd5c9f455-cmh2h\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") " pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:09.161510 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:09.161450    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:09.612730 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:09.612706    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"]
Oct 30 23:49:09.614977 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:49:09.614952    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdf11a9b_5acc_40dd_86e8_8fde0cd486d3.slice/crio-05a5eb170e8dc5ee7fda9c501a4279b69a42c21995af35f22fbb29b0a9325c47 WatchSource:0}: Error finding container 05a5eb170e8dc5ee7fda9c501a4279b69a42c21995af35f22fbb29b0a9325c47: Status 404 returned error can't find the container with id 05a5eb170e8dc5ee7fda9c501a4279b69a42c21995af35f22fbb29b0a9325c47
Oct 30 23:49:09.856586 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:09.856543    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h" event={"ID":"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3","Type":"ContainerStarted","Data":"82e132c2cfbc1284fc710e6ee11db1236cb40ed034244fd1d71d02dbb294c02b"}
Oct 30 23:49:09.857379 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:09.856631    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h" event={"ID":"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3","Type":"ContainerStarted","Data":"05a5eb170e8dc5ee7fda9c501a4279b69a42c21995af35f22fbb29b0a9325c47"}
Oct 30 23:49:09.972884 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:09.972788    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h" podStartSLOduration=1.9727743979999999 podStartE2EDuration="1.972774398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:09.968740861 +0000 UTC m=+2535.856647217" watchObservedRunningTime="2026-10-30 23:49:09.972774398 +0000 UTC m=+2535.860680751"
Oct 30 23:49:10.162275 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:10.162245    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:17.592527 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:49:17.592494    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:49:19.162492 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:19.162458    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:19.175547 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:19.175515    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:19.942482 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:19.942398    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:49:22.194478 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.194445    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"]
Oct 30 23:49:22.203157 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.203128    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.272623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.265951    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-333b59-openshift-default-dockercfg-9j7tz\""
Oct 30 23:49:22.393943 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.393902    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-workload-socket\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.394206 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.394188    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-token\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.394392 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.394374    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-envoy\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.399641 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.394540    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357e99ba-4145-479f-aa06-cf148c2c52b4-istiod-ca-cert\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.399937 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.399912    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-credential-socket\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.400112 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.400097    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-workload-certs\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.400223 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.400208    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-data\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.400364 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.400349    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb9xv\" (UniqueName: \"kubernetes.io/projected/357e99ba-4145-479f-aa06-cf148c2c52b4-kube-api-access-gb9xv\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.400525 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.400510    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-podinfo\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.490459 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.490387    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"]
Oct 30 23:49:22.502246 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.502218    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-workload-certs\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.502425 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.502252    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-data\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.502425 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.502283    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb9xv\" (UniqueName: \"kubernetes.io/projected/357e99ba-4145-479f-aa06-cf148c2c52b4-kube-api-access-gb9xv\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.502425 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.502345    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-podinfo\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.502425 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.502392    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-workload-socket\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.502425 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.502413    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-token\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.502738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.502480    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-envoy\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.502738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.502530    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357e99ba-4145-479f-aa06-cf148c2c52b4-istiod-ca-cert\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.502738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.502551    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-credential-socket\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.503304 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.503074    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-workload-socket\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.503405 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.503313    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-workload-certs\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.503874 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.503550    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-data\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.503874 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.503838    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-credential-socket\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.504038 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.503898    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357e99ba-4145-479f-aa06-cf148c2c52b4-istiod-ca-cert\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.505947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.505896    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-envoy\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.506378 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.506358    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-podinfo\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.566341 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.566310    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb9xv\" (UniqueName: \"kubernetes.io/projected/357e99ba-4145-479f-aa06-cf148c2c52b4-kube-api-access-gb9xv\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.566482 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.566362    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-token\") pod \"e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") " pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:22.693833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:22.693801    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:23.187776 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:23.187750    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"]
Oct 30 23:49:23.188739 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:49:23.188714    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod357e99ba_4145_479f_aa06_cf148c2c52b4.slice/crio-3be08087284723c18d0fd85cb2350f31a8b1c73c4efa23455ebfb416d67fe776 WatchSource:0}: Error finding container 3be08087284723c18d0fd85cb2350f31a8b1c73c4efa23455ebfb416d67fe776: Status 404 returned error can't find the container with id 3be08087284723c18d0fd85cb2350f31a8b1c73c4efa23455ebfb416d67fe776
Oct 30 23:49:23.191304 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:23.191276    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:49:23.191387 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:23.191342    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:49:23.191387 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:23.191374    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:49:23.962774 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:23.962743    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q" event={"ID":"357e99ba-4145-479f-aa06-cf148c2c52b4","Type":"ContainerStarted","Data":"4649efa82104f0303deeedb98930ecf6645f625e240e42b5724dee27b0dcc834"}
Oct 30 23:49:23.963210 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:23.962792    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q" event={"ID":"357e99ba-4145-479f-aa06-cf148c2c52b4","Type":"ContainerStarted","Data":"3be08087284723c18d0fd85cb2350f31a8b1c73c4efa23455ebfb416d67fe776"}
Oct 30 23:49:24.128977 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:24.128911    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q" podStartSLOduration=2.128893429 podStartE2EDuration="2.128893429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:49:24.128263937 +0000 UTC m=+2550.016170291" watchObservedRunningTime="2026-10-30 23:49:24.128893429 +0000 UTC m=+2550.016799782"
Oct 30 23:49:24.694472 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:24.694441    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:24.700127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:24.700096    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:24.969159 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:24.969081    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:24.970040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:24.970023    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:49:28.357116 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:28.357079    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc"]
Oct 30 23:49:28.406841 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:28.406804    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-3c9613-86zxc"]
Oct 30 23:49:30.299124 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:30.299078    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd" path="/var/lib/kubelet/pods/cf0fd94f-6aef-4c8d-9b79-4d29cb6437cd/volumes"
Oct 30 23:49:35.366675 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:35.366641    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"]
Oct 30 23:49:35.367168 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:35.366904    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j" podUID="7136a40b-0373-4850-81c8-1345ff2b8996" containerName="istio-proxy" containerID="cri-o://e08cd65627b38fd7d88b3c421495513b09555f483fa58caf3bb3ca7758385df3" gracePeriod=30
Oct 30 23:49:40.648439 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.648373    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:49:40.710007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.709981    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7136a40b-0373-4850-81c8-1345ff2b8996-istio-token\" (UniqueName: \"kubernetes.io/projected/7136a40b-0373-4850-81c8-1345ff2b8996-istio-token\") pod \"7136a40b-0373-4850-81c8-1345ff2b8996\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") "
Oct 30 23:49:40.710233 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.710022    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-workload-socket\") pod \"7136a40b-0373-4850-81c8-1345ff2b8996\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") "
Oct 30 23:49:40.710233 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.710042    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-workload-certs\") pod \"7136a40b-0373-4850-81c8-1345ff2b8996\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") "
Oct 30 23:49:40.710233 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.710111    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7136a40b-0373-4850-81c8-1345ff2b8996-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7136a40b-0373-4850-81c8-1345ff2b8996-istio-podinfo\") pod \"7136a40b-0373-4850-81c8-1345ff2b8996\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") "
Oct 30 23:49:40.710233 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.710156    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-istio-envoy\") pod \"7136a40b-0373-4850-81c8-1345ff2b8996\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") "
Oct 30 23:49:40.710233 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.710202    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-istio-data\") pod \"7136a40b-0373-4850-81c8-1345ff2b8996\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") "
Oct 30 23:49:40.710639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.710313    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7136a40b-0373-4850-81c8-1345ff2b8996-kube-api-access-22fhn\" (UniqueName: \"kubernetes.io/projected/7136a40b-0373-4850-81c8-1345ff2b8996-kube-api-access-22fhn\") pod \"7136a40b-0373-4850-81c8-1345ff2b8996\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") "
Oct 30 23:49:40.710639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.710341    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7136a40b-0373-4850-81c8-1345ff2b8996-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7136a40b-0373-4850-81c8-1345ff2b8996-istiod-ca-cert\") pod \"7136a40b-0373-4850-81c8-1345ff2b8996\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") "
Oct 30 23:49:40.710639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.710391    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-credential-socket\") pod \"7136a40b-0373-4850-81c8-1345ff2b8996\" (UID: \"7136a40b-0373-4850-81c8-1345ff2b8996\") "
Oct 30 23:49:40.710639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.710320    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-workload-socket" pod "7136a40b-0373-4850-81c8-1345ff2b8996" (UID: "7136a40b-0373-4850-81c8-1345ff2b8996"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:40.710639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.710460    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-workload-certs" pod "7136a40b-0373-4850-81c8-1345ff2b8996" (UID: "7136a40b-0373-4850-81c8-1345ff2b8996"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:40.710639 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.710553    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-istio-data" pod "7136a40b-0373-4850-81c8-1345ff2b8996" (UID: "7136a40b-0373-4850-81c8-1345ff2b8996"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:40.710989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.710798    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7136a40b-0373-4850-81c8-1345ff2b8996-istiod-ca-cert" pod "7136a40b-0373-4850-81c8-1345ff2b8996" (UID: "7136a40b-0373-4850-81c8-1345ff2b8996"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:49:40.710989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.710808    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-credential-socket" pod "7136a40b-0373-4850-81c8-1345ff2b8996" (UID: "7136a40b-0373-4850-81c8-1345ff2b8996"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:40.711093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.711008    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:49:40.711093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.711028    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:49:40.711093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.711041    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:49:40.711093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.711055    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7136a40b-0373-4850-81c8-1345ff2b8996-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:49:40.711093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.711068    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:49:40.712457 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.712434    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7136a40b-0373-4850-81c8-1345ff2b8996-istio-token" pod "7136a40b-0373-4850-81c8-1345ff2b8996" (UID: "7136a40b-0373-4850-81c8-1345ff2b8996"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:49:40.712886 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.712870    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-istio-envoy" pod "7136a40b-0373-4850-81c8-1345ff2b8996" (UID: "7136a40b-0373-4850-81c8-1345ff2b8996"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:40.713206 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.713190    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7136a40b-0373-4850-81c8-1345ff2b8996-istio-podinfo" pod "7136a40b-0373-4850-81c8-1345ff2b8996" (UID: "7136a40b-0373-4850-81c8-1345ff2b8996"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:49:40.713206 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.713197    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7136a40b-0373-4850-81c8-1345ff2b8996-kube-api-access-22fhn" pod "7136a40b-0373-4850-81c8-1345ff2b8996" (UID: "7136a40b-0373-4850-81c8-1345ff2b8996"). InnerVolumeSpecName "kube-api-access-22fhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:49:40.812508 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.812478    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7136a40b-0373-4850-81c8-1345ff2b8996-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:49:40.812508 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.812509    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7136a40b-0373-4850-81c8-1345ff2b8996-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:49:40.812736 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.812525    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7136a40b-0373-4850-81c8-1345ff2b8996-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:49:40.812736 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:40.812535    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22fhn\" (UniqueName: \"kubernetes.io/projected/7136a40b-0373-4850-81c8-1345ff2b8996-kube-api-access-22fhn\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:49:41.185582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:41.185547    5078 generic.go:356] "Generic (PLEG): container finished" podID="7136a40b-0373-4850-81c8-1345ff2b8996" containerID="e08cd65627b38fd7d88b3c421495513b09555f483fa58caf3bb3ca7758385df3" exitCode=0
Oct 30 23:49:41.185804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:41.185619    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"
Oct 30 23:49:41.185804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:41.185622    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j" event={"ID":"7136a40b-0373-4850-81c8-1345ff2b8996","Type":"ContainerDied","Data":"e08cd65627b38fd7d88b3c421495513b09555f483fa58caf3bb3ca7758385df3"}
Oct 30 23:49:41.185804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:41.185672    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j" event={"ID":"7136a40b-0373-4850-81c8-1345ff2b8996","Type":"ContainerDied","Data":"177e42a7043fa2ffc51b42a02fcfa157be25934f88c87a27c5491cdc896790b5"}
Oct 30 23:49:41.185804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:41.185695    5078 scope.go:122] "RemoveContainer" containerID="e08cd65627b38fd7d88b3c421495513b09555f483fa58caf3bb3ca7758385df3"
Oct 30 23:49:41.197047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:41.197028    5078 scope.go:122] "RemoveContainer" containerID="1aa37ceb910cc0810fffe763e4a8a06bc4a6f437b92e79d2899ffd76445d96fb"
Oct 30 23:49:41.205827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:41.205810    5078 scope.go:122] "RemoveContainer" containerID="e08cd65627b38fd7d88b3c421495513b09555f483fa58caf3bb3ca7758385df3"
Oct 30 23:49:41.206103 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:49:41.206073    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e08cd65627b38fd7d88b3c421495513b09555f483fa58caf3bb3ca7758385df3\": container with ID starting with e08cd65627b38fd7d88b3c421495513b09555f483fa58caf3bb3ca7758385df3 not found: ID does not exist" containerID="e08cd65627b38fd7d88b3c421495513b09555f483fa58caf3bb3ca7758385df3"
Oct 30 23:49:41.206171 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:41.206102    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e08cd65627b38fd7d88b3c421495513b09555f483fa58caf3bb3ca7758385df3"} err="failed to get container status \"e08cd65627b38fd7d88b3c421495513b09555f483fa58caf3bb3ca7758385df3\": rpc error: code = NotFound desc = could not find container \"e08cd65627b38fd7d88b3c421495513b09555f483fa58caf3bb3ca7758385df3\": container with ID starting with e08cd65627b38fd7d88b3c421495513b09555f483fa58caf3bb3ca7758385df3 not found: ID does not exist"
Oct 30 23:49:41.206171 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:41.206122    5078 scope.go:122] "RemoveContainer" containerID="1aa37ceb910cc0810fffe763e4a8a06bc4a6f437b92e79d2899ffd76445d96fb"
Oct 30 23:49:41.206365 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:49:41.206348    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aa37ceb910cc0810fffe763e4a8a06bc4a6f437b92e79d2899ffd76445d96fb\": container with ID starting with 1aa37ceb910cc0810fffe763e4a8a06bc4a6f437b92e79d2899ffd76445d96fb not found: ID does not exist" containerID="1aa37ceb910cc0810fffe763e4a8a06bc4a6f437b92e79d2899ffd76445d96fb"
Oct 30 23:49:41.206412 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:41.206371    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aa37ceb910cc0810fffe763e4a8a06bc4a6f437b92e79d2899ffd76445d96fb"} err="failed to get container status \"1aa37ceb910cc0810fffe763e4a8a06bc4a6f437b92e79d2899ffd76445d96fb\": rpc error: code = NotFound desc = could not find container \"1aa37ceb910cc0810fffe763e4a8a06bc4a6f437b92e79d2899ffd76445d96fb\": container with ID starting with 1aa37ceb910cc0810fffe763e4a8a06bc4a6f437b92e79d2899ffd76445d96fb not found: ID does not exist"
Oct 30 23:49:41.350675 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:41.350648    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"]
Oct 30 23:49:41.400799 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:41.400771    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-3c9613-openshift-default-769cf65595-52p8j"]
Oct 30 23:49:42.299092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:42.299058    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7136a40b-0373-4850-81c8-1345ff2b8996" path="/var/lib/kubelet/pods/7136a40b-0373-4850-81c8-1345ff2b8996/volumes"
Oct 30 23:49:47.591809 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:49:47.591774    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:49:52.875087 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:52.874954    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"]
Oct 30 23:49:52.875635 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:52.875258    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n" podUID="26dfc3f5-4b7f-4524-bd85-18df3da28f8d" containerName="istio-proxy" containerID="cri-o://cfb9c21ce59bb3bfe5495e4d4199165f1fd58ae11f96414a97f5dd3955ce3a3b" gracePeriod=30
Oct 30 23:49:54.004254 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:54.004222    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:49:54.004654 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:54.004276    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n" podUID="26dfc3f5-4b7f-4524-bd85-18df3da28f8d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:49:58.139544 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.139479    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:49:58.237213 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.237187    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-data\") pod \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") "
Oct 30 23:49:58.237390 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.237240    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-podinfo\") pod \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") "
Oct 30 23:49:58.237390 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.237259    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-token\" (UniqueName: \"kubernetes.io/projected/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-token\") pod \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") "
Oct 30 23:49:58.237390 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.237372    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-envoy\") pod \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") "
Oct 30 23:49:58.237584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.237426    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-workload-certs\") pod \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") "
Oct 30 23:49:58.237584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.237447    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-data" pod "26dfc3f5-4b7f-4524-bd85-18df3da28f8d" (UID: "26dfc3f5-4b7f-4524-bd85-18df3da28f8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:58.237584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.237524    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-workload-socket\") pod \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") "
Oct 30 23:49:58.237806 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.237586    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istiod-ca-cert\") pod \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") "
Oct 30 23:49:58.237806 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.237710    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-kube-api-access-tr4km\" (UniqueName: \"kubernetes.io/projected/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-kube-api-access-tr4km\") pod \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") "
Oct 30 23:49:58.237806 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.237756    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-credential-socket\") pod \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\" (UID: \"26dfc3f5-4b7f-4524-bd85-18df3da28f8d\") "
Oct 30 23:49:58.237806 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.237769    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-workload-certs" pod "26dfc3f5-4b7f-4524-bd85-18df3da28f8d" (UID: "26dfc3f5-4b7f-4524-bd85-18df3da28f8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:58.238022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.237933    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istiod-ca-cert" pod "26dfc3f5-4b7f-4524-bd85-18df3da28f8d" (UID: "26dfc3f5-4b7f-4524-bd85-18df3da28f8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:49:58.238244 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.238082    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-credential-socket" pod "26dfc3f5-4b7f-4524-bd85-18df3da28f8d" (UID: "26dfc3f5-4b7f-4524-bd85-18df3da28f8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:58.238244 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.238130    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-workload-socket" pod "26dfc3f5-4b7f-4524-bd85-18df3da28f8d" (UID: "26dfc3f5-4b7f-4524-bd85-18df3da28f8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:58.238365 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.238298    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:49:58.238365 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.238311    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:49:58.238365 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.238319    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:49:58.238365 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.238328    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:49:58.238365 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.238337    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:49:58.239787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.239761    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-token" pod "26dfc3f5-4b7f-4524-bd85-18df3da28f8d" (UID: "26dfc3f5-4b7f-4524-bd85-18df3da28f8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:49:58.239787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.239775    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-envoy" pod "26dfc3f5-4b7f-4524-bd85-18df3da28f8d" (UID: "26dfc3f5-4b7f-4524-bd85-18df3da28f8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:49:58.240052 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.240035    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-kube-api-access-tr4km" pod "26dfc3f5-4b7f-4524-bd85-18df3da28f8d" (UID: "26dfc3f5-4b7f-4524-bd85-18df3da28f8d"). InnerVolumeSpecName "kube-api-access-tr4km". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:49:58.240199 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.240176    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-podinfo" pod "26dfc3f5-4b7f-4524-bd85-18df3da28f8d" (UID: "26dfc3f5-4b7f-4524-bd85-18df3da28f8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:49:58.298511 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.298474    5078 generic.go:356] "Generic (PLEG): container finished" podID="26dfc3f5-4b7f-4524-bd85-18df3da28f8d" containerID="cfb9c21ce59bb3bfe5495e4d4199165f1fd58ae11f96414a97f5dd3955ce3a3b" exitCode=0
Oct 30 23:49:58.298694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.298508    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n" event={"ID":"26dfc3f5-4b7f-4524-bd85-18df3da28f8d","Type":"ContainerDied","Data":"cfb9c21ce59bb3bfe5495e4d4199165f1fd58ae11f96414a97f5dd3955ce3a3b"}
Oct 30 23:49:58.298694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.298559    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n" event={"ID":"26dfc3f5-4b7f-4524-bd85-18df3da28f8d","Type":"ContainerDied","Data":"7548176ec59ade9a5c235c9d6b75dd81ae90a801b50e5df85b1feb6c94949b99"}
Oct 30 23:49:58.298694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.298566    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"
Oct 30 23:49:58.298694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.298589    5078 scope.go:122] "RemoveContainer" containerID="cfb9c21ce59bb3bfe5495e4d4199165f1fd58ae11f96414a97f5dd3955ce3a3b"
Oct 30 23:49:58.317018 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.316996    5078 scope.go:122] "RemoveContainer" containerID="cfb9c21ce59bb3bfe5495e4d4199165f1fd58ae11f96414a97f5dd3955ce3a3b"
Oct 30 23:49:58.317377 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:49:58.317357    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfb9c21ce59bb3bfe5495e4d4199165f1fd58ae11f96414a97f5dd3955ce3a3b\": container with ID starting with cfb9c21ce59bb3bfe5495e4d4199165f1fd58ae11f96414a97f5dd3955ce3a3b not found: ID does not exist" containerID="cfb9c21ce59bb3bfe5495e4d4199165f1fd58ae11f96414a97f5dd3955ce3a3b"
Oct 30 23:49:58.317467 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.317391    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfb9c21ce59bb3bfe5495e4d4199165f1fd58ae11f96414a97f5dd3955ce3a3b"} err="failed to get container status \"cfb9c21ce59bb3bfe5495e4d4199165f1fd58ae11f96414a97f5dd3955ce3a3b\": rpc error: code = NotFound desc = could not find container \"cfb9c21ce59bb3bfe5495e4d4199165f1fd58ae11f96414a97f5dd3955ce3a3b\": container with ID starting with cfb9c21ce59bb3bfe5495e4d4199165f1fd58ae11f96414a97f5dd3955ce3a3b not found: ID does not exist"
Oct 30 23:49:58.339314 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.339280    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr4km\" (UniqueName: \"kubernetes.io/projected/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-kube-api-access-tr4km\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:49:58.339314 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.339312    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:49:58.339510 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.339323    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:49:58.339510 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.339332    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26dfc3f5-4b7f-4524-bd85-18df3da28f8d-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:49:58.471928 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.471834    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"]
Oct 30 23:49:58.477432 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:49:58.477404    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-rwt8n"]
Oct 30 23:50:00.079478 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.079449    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"]
Oct 30 23:50:00.079996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.079980    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7136a40b-0373-4850-81c8-1345ff2b8996" containerName="istio-proxy"
Oct 30 23:50:00.080043 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.079997    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7136a40b-0373-4850-81c8-1345ff2b8996" containerName="istio-proxy"
Oct 30 23:50:00.080043 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.080019    5078 cpu_manager.go:414] "containerMap: removing container" podUID="26dfc3f5-4b7f-4524-bd85-18df3da28f8d" containerName="istio-proxy"
Oct 30 23:50:00.080043 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.080024    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26dfc3f5-4b7f-4524-bd85-18df3da28f8d" containerName="istio-proxy"
Oct 30 23:50:00.080043 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.080042    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7136a40b-0373-4850-81c8-1345ff2b8996" containerName="istio-proxy"
Oct 30 23:50:00.080184 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.080047    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7136a40b-0373-4850-81c8-1345ff2b8996" containerName="istio-proxy"
Oct 30 23:50:00.080184 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.080134    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7136a40b-0373-4850-81c8-1345ff2b8996" containerName="istio-proxy"
Oct 30 23:50:00.080184 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.080146    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="26dfc3f5-4b7f-4524-bd85-18df3da28f8d" containerName="istio-proxy"
Oct 30 23:50:00.084711 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.084694    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.187679 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.187644    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-15b137\"/\"test-model-15b137-kserve-self-signed-certs\""
Oct 30 23:50:00.187679 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.187652    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-15b137\"/\"kube-root-ca.crt\""
Oct 30 23:50:00.187921 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.187740    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-15b137\"/\"default-dockercfg-x4qhc\""
Oct 30 23:50:00.187921 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.187747    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-15b137\"/\"openshift-service-ca.crt\""
Oct 30 23:50:00.191130 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.191108    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"]
Oct 30 23:50:00.264623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.263646    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-tmp-dir\") pod \"test-model-15b137-kserve-6fb8db7765-t8rs7\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") " pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.264623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.263783    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-model-cache\") pod \"test-model-15b137-kserve-6fb8db7765-t8rs7\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") " pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.264623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.263831    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-home\") pod \"test-model-15b137-kserve-6fb8db7765-t8rs7\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") " pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.264623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.263868    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-tls-certs\") pod \"test-model-15b137-kserve-6fb8db7765-t8rs7\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") " pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.264623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.264028    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-dshm\") pod \"test-model-15b137-kserve-6fb8db7765-t8rs7\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") " pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.264623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.264110    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qds4\" (UniqueName: \"kubernetes.io/projected/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-kube-api-access-4qds4\") pod \"test-model-15b137-kserve-6fb8db7765-t8rs7\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") " pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.299121 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.299090    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26dfc3f5-4b7f-4524-bd85-18df3da28f8d" path="/var/lib/kubelet/pods/26dfc3f5-4b7f-4524-bd85-18df3da28f8d/volumes"
Oct 30 23:50:00.365001 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.364918    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qds4\" (UniqueName: \"kubernetes.io/projected/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-kube-api-access-4qds4\") pod \"test-model-15b137-kserve-6fb8db7765-t8rs7\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") " pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.365001 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.364963    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-tmp-dir\") pod \"test-model-15b137-kserve-6fb8db7765-t8rs7\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") " pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.365207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.365040    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-model-cache\") pod \"test-model-15b137-kserve-6fb8db7765-t8rs7\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") " pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.365207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.365092    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-home\") pod \"test-model-15b137-kserve-6fb8db7765-t8rs7\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") " pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.365207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.365120    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-tls-certs\") pod \"test-model-15b137-kserve-6fb8db7765-t8rs7\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") " pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.365373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.365268    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-dshm\") pod \"test-model-15b137-kserve-6fb8db7765-t8rs7\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") " pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.365492 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.365472    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-tmp-dir\") pod \"test-model-15b137-kserve-6fb8db7765-t8rs7\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") " pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.365571 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.365489    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-model-cache\") pod \"test-model-15b137-kserve-6fb8db7765-t8rs7\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") " pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.365571 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.365529    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-home\") pod \"test-model-15b137-kserve-6fb8db7765-t8rs7\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") " pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.367675 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.367646    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-dshm\") pod \"test-model-15b137-kserve-6fb8db7765-t8rs7\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") " pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.367925 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.367911    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-tls-certs\") pod \"test-model-15b137-kserve-6fb8db7765-t8rs7\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") " pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.423987 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.423963    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qds4\" (UniqueName: \"kubernetes.io/projected/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-kube-api-access-4qds4\") pod \"test-model-15b137-kserve-6fb8db7765-t8rs7\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") " pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:00.696885 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:00.696788    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:01.047613 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:01.047570    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8"]
Oct 30 23:50:01.063896 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:01.063868    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b9d2863-gwll8"]
Oct 30 23:50:01.097432 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:01.097399    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"]
Oct 30 23:50:01.114342 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:50:01.114318    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96a3c9e3_edeb_4f10_8b80_7d42fd84b3dc.slice/crio-002fb914e98585ee9f8306283b4b4023139b418105d2670ffc1806196604eb42 WatchSource:0}: Error finding container 002fb914e98585ee9f8306283b4b4023139b418105d2670ffc1806196604eb42: Status 404 returned error can't find the container with id 002fb914e98585ee9f8306283b4b4023139b418105d2670ffc1806196604eb42
Oct 30 23:50:01.320576 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:01.320493    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7" event={"ID":"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc","Type":"ContainerStarted","Data":"f60fe5a37e7e36572a2e74a483548b1aeb7db696fb3f97805b96b77ad2ffb1ae"}
Oct 30 23:50:01.320576 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:01.320534    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7" event={"ID":"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc","Type":"ContainerStarted","Data":"002fb914e98585ee9f8306283b4b4023139b418105d2670ffc1806196604eb42"}
Oct 30 23:50:01.435613 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:01.435545    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7" podStartSLOduration=1.435531123 podStartE2EDuration="1.435531123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:50:01.43352963 +0000 UTC m=+2587.321435983" watchObservedRunningTime="2026-10-30 23:50:01.435531123 +0000 UTC m=+2587.323437453"
Oct 30 23:50:01.697805 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:01.697662    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:02.046901 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:02.046863    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd"]
Oct 30 23:50:02.096627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:02.096581    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad4ef5bb-zlwzd"]
Oct 30 23:50:02.298317 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:02.298245    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cfecc2c-1a44-4f1d-828f-69f08b03a3f4" path="/var/lib/kubelet/pods/7cfecc2c-1a44-4f1d-828f-69f08b03a3f4/volumes"
Oct 30 23:50:02.298710 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:02.298687    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b96c7b9a-2055-4654-b26c-d370869a2f1d" path="/var/lib/kubelet/pods/b96c7b9a-2055-4654-b26c-d370869a2f1d/volumes"
Oct 30 23:50:03.448605 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:03.448573    5078 scope.go:122] "RemoveContainer" containerID="d574d219b312901b4d05007261161e594ea137637924edb53d7682b33987e977"
Oct 30 23:50:03.458143 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:03.458116    5078 scope.go:122] "RemoveContainer" containerID="93b85680f764963017b24c6d10472f57733b70e268462c3d0967520b0c35ce05"
Oct 30 23:50:10.697259 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:10.697224    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:10.709803 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:10.709779    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:11.399897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:11.399869    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:50:17.591653 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:50:17.591617    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:50:34.048570 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:34.048485    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb"]
Oct 30 23:50:34.098376 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:34.098341    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8"]
Oct 30 23:50:34.101156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:34.101125    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f7a9791-44mlb"]
Oct 30 23:50:34.103384 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:34.103362    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-984b7668-d89h8"]
Oct 30 23:50:34.299180 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:34.299086    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a07d752-751e-4a29-8a8d-f23d2144b3ba" path="/var/lib/kubelet/pods/3a07d752-751e-4a29-8a8d-f23d2144b3ba/volumes"
Oct 30 23:50:34.299686 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:34.299663    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b" path="/var/lib/kubelet/pods/a7ab1f5f-0fab-42cc-8e82-178a0cbcc07b/volumes"
Oct 30 23:50:47.593808 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:50:47.593765    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:50:56.754203 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:56.754168    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw"]
Oct 30 23:50:56.803536 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:56.803494    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-505c1f-q6jbw"]
Oct 30 23:50:58.298328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:50:58.298285    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a2549ce-4cdf-4965-917e-957a14cd7265" path="/var/lib/kubelet/pods/7a2549ce-4cdf-4965-917e-957a14cd7265/volumes"
Oct 30 23:51:03.497622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:03.495432    5078 scope.go:122] "RemoveContainer" containerID="cdfadb54f6a0e13ed6806256c77309d613624dd94d4587d67502c0111a9d4840"
Oct 30 23:51:03.507575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:03.507550    5078 scope.go:122] "RemoveContainer" containerID="2a3229f2f839a4452a217f153dd1e28fc700f68dfcb59fe36853225fa85a6d0f"
Oct 30 23:51:03.517782 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:03.517759    5078 scope.go:122] "RemoveContainer" containerID="97367d30d0aab917ec6fc6073b9a509ab30f319ea5efe616d63d647ecbd364e0"
Oct 30 23:51:03.526882 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:03.526860    5078 scope.go:122] "RemoveContainer" containerID="adb018a3323611ec6c9ac7d78d378b2698547915bd131a24e00e628605e9e23d"
Oct 30 23:51:03.536934 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:03.536899    5078 scope.go:122] "RemoveContainer" containerID="1655d3c31cdfef6416e344ce6ad55cd0fdd03a5fd106132adf00815f85cd053b"
Oct 30 23:51:04.138105 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.138076    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn"]
Oct 30 23:51:04.138690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.138674    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7136a40b-0373-4850-81c8-1345ff2b8996" containerName="istio-proxy"
Oct 30 23:51:04.143462 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.143445    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn"
Oct 30 23:51:04.199553 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.199513    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn"]
Oct 30 23:51:04.244938 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.244905    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:51:04.244938 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.244920    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:51:04.245166 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.245008    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:51:04.245812 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.245786    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:51:04.288582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.288551    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867eeb4a-d196-4cde-b5ca-52145828f464-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn\" (UID: \"867eeb4a-d196-4cde-b5ca-52145828f464\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn"
Oct 30 23:51:04.288782 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.288727    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867eeb4a-d196-4cde-b5ca-52145828f464-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn\" (UID: \"867eeb4a-d196-4cde-b5ca-52145828f464\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn"
Oct 30 23:51:04.288782 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.288775    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txs5r\" (UniqueName: \"kubernetes.io/projected/867eeb4a-d196-4cde-b5ca-52145828f464-kube-api-access-txs5r\") pod \"maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn\" (UID: \"867eeb4a-d196-4cde-b5ca-52145828f464\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn"
Oct 30 23:51:04.389888 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.389796    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867eeb4a-d196-4cde-b5ca-52145828f464-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn\" (UID: \"867eeb4a-d196-4cde-b5ca-52145828f464\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn"
Oct 30 23:51:04.390058 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.389950    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867eeb4a-d196-4cde-b5ca-52145828f464-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn\" (UID: \"867eeb4a-d196-4cde-b5ca-52145828f464\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn"
Oct 30 23:51:04.390058 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.390015    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txs5r\" (UniqueName: \"kubernetes.io/projected/867eeb4a-d196-4cde-b5ca-52145828f464-kube-api-access-txs5r\") pod \"maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn\" (UID: \"867eeb4a-d196-4cde-b5ca-52145828f464\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn"
Oct 30 23:51:04.390633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.390611    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867eeb4a-d196-4cde-b5ca-52145828f464-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn\" (UID: \"867eeb4a-d196-4cde-b5ca-52145828f464\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn"
Oct 30 23:51:04.392476 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.392458    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867eeb4a-d196-4cde-b5ca-52145828f464-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn\" (UID: \"867eeb4a-d196-4cde-b5ca-52145828f464\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn"
Oct 30 23:51:04.447525 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.447496    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txs5r\" (UniqueName: \"kubernetes.io/projected/867eeb4a-d196-4cde-b5ca-52145828f464-kube-api-access-txs5r\") pod \"maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn\" (UID: \"867eeb4a-d196-4cde-b5ca-52145828f464\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn"
Oct 30 23:51:04.457496 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.457469    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn"
Oct 30 23:51:04.834179 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:04.834150    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn"]
Oct 30 23:51:04.836026 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:51:04.835998    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867eeb4a_d196_4cde_b5ca_52145828f464.slice/crio-2d4fda72022d9b1b0746522753ebd63765375d54572e3f037227bfd90d0110bd WatchSource:0}: Error finding container 2d4fda72022d9b1b0746522753ebd63765375d54572e3f037227bfd90d0110bd: Status 404 returned error can't find the container with id 2d4fda72022d9b1b0746522753ebd63765375d54572e3f037227bfd90d0110bd
Oct 30 23:51:05.771968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:05.771937    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn" event={"ID":"867eeb4a-d196-4cde-b5ca-52145828f464","Type":"ContainerStarted","Data":"5ea64bce9a9cdb0bab67abe5e3d6ef1d19a0a5b8f9a5fd006fb5312e68409372"}
Oct 30 23:51:05.772165 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:05.771988    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn" event={"ID":"867eeb4a-d196-4cde-b5ca-52145828f464","Type":"ContainerStarted","Data":"2d4fda72022d9b1b0746522753ebd63765375d54572e3f037227bfd90d0110bd"}
Oct 30 23:51:05.895293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:05.895238    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn" podStartSLOduration=1.895221597 podStartE2EDuration="1.895221597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:05.892787812 +0000 UTC m=+2651.780694165" watchObservedRunningTime="2026-10-30 23:51:05.895221597 +0000 UTC m=+2651.783127983"
Oct 30 23:51:06.781467 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:06.781434    5078 generic.go:356] "Generic (PLEG): container finished" podID="867eeb4a-d196-4cde-b5ca-52145828f464" containerID="5ea64bce9a9cdb0bab67abe5e3d6ef1d19a0a5b8f9a5fd006fb5312e68409372" exitCode=0
Oct 30 23:51:06.781660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:06.781496    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn" event={"ID":"867eeb4a-d196-4cde-b5ca-52145828f464","Type":"ContainerDied","Data":"5ea64bce9a9cdb0bab67abe5e3d6ef1d19a0a5b8f9a5fd006fb5312e68409372"}
Oct 30 23:51:07.960817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:07.960791    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn"
Oct 30 23:51:08.047167 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:08.047132    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/867eeb4a-d196-4cde-b5ca-52145828f464-kube-api-access-txs5r\" (UniqueName: \"kubernetes.io/projected/867eeb4a-d196-4cde-b5ca-52145828f464-kube-api-access-txs5r\") pod \"867eeb4a-d196-4cde-b5ca-52145828f464\" (UID: \"867eeb4a-d196-4cde-b5ca-52145828f464\") "
Oct 30 23:51:08.047333 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:08.047188    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/867eeb4a-d196-4cde-b5ca-52145828f464-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867eeb4a-d196-4cde-b5ca-52145828f464-maas-api-openshift-ca\") pod \"867eeb4a-d196-4cde-b5ca-52145828f464\" (UID: \"867eeb4a-d196-4cde-b5ca-52145828f464\") "
Oct 30 23:51:08.047333 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:08.047319    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/867eeb4a-d196-4cde-b5ca-52145828f464-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867eeb4a-d196-4cde-b5ca-52145828f464-maas-api-xks-ca\") pod \"867eeb4a-d196-4cde-b5ca-52145828f464\" (UID: \"867eeb4a-d196-4cde-b5ca-52145828f464\") "
Oct 30 23:51:08.047539 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:08.047509    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/867eeb4a-d196-4cde-b5ca-52145828f464-maas-api-openshift-ca" pod "867eeb4a-d196-4cde-b5ca-52145828f464" (UID: "867eeb4a-d196-4cde-b5ca-52145828f464"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:51:08.047834 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:08.047817    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/867eeb4a-d196-4cde-b5ca-52145828f464-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:51:08.049497 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:08.049476    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/867eeb4a-d196-4cde-b5ca-52145828f464-maas-api-xks-ca" pod "867eeb4a-d196-4cde-b5ca-52145828f464" (UID: "867eeb4a-d196-4cde-b5ca-52145828f464"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:51:08.049645 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:08.049628    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867eeb4a-d196-4cde-b5ca-52145828f464-kube-api-access-txs5r" pod "867eeb4a-d196-4cde-b5ca-52145828f464" (UID: "867eeb4a-d196-4cde-b5ca-52145828f464"). InnerVolumeSpecName "kube-api-access-txs5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:51:08.148881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:08.148842    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txs5r\" (UniqueName: \"kubernetes.io/projected/867eeb4a-d196-4cde-b5ca-52145828f464-kube-api-access-txs5r\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:51:08.148881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:08.148876    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/867eeb4a-d196-4cde-b5ca-52145828f464-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:51:08.795214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:08.795187    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn"
Oct 30 23:51:08.795214 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:08.795207    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn" event={"ID":"867eeb4a-d196-4cde-b5ca-52145828f464","Type":"ContainerDied","Data":"2d4fda72022d9b1b0746522753ebd63765375d54572e3f037227bfd90d0110bd"}
Oct 30 23:51:08.795435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:08.795260    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d4fda72022d9b1b0746522753ebd63765375d54572e3f037227bfd90d0110bd"
Oct 30 23:51:15.729390 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:15.729356    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm"]
Oct 30 23:51:15.844040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:15.844002    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv"]
Oct 30 23:51:17.615507 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:51:17.615472    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:51:20.730709 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:20.730660    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm" podUID="de020d7e-ce62-476f-9ce1-1ecd344f05b8" containerName="payload-processing" containerID="cri-o://7cd2dd1e56af7184ef60d8d45fe5816f29872033898cb17051e00a3b31465050" gracePeriod=25
Oct 30 23:51:20.845398 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:20.845362    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv" podUID="b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8" containerName="payload-pre-processing" containerID="cri-o://0f1b58d42637429fef957c46f4df3331a96128ffe244912612fbbeece223ba3a" gracePeriod=25
Oct 30 23:51:20.882777 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:20.882749    5078 generic.go:356] "Generic (PLEG): container finished" podID="de020d7e-ce62-476f-9ce1-1ecd344f05b8" containerID="7cd2dd1e56af7184ef60d8d45fe5816f29872033898cb17051e00a3b31465050" exitCode=0
Oct 30 23:51:20.882929 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:20.882827    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm" event={"ID":"de020d7e-ce62-476f-9ce1-1ecd344f05b8","Type":"ContainerDied","Data":"7cd2dd1e56af7184ef60d8d45fe5816f29872033898cb17051e00a3b31465050"}
Oct 30 23:51:20.884498 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:20.884478    5078 generic.go:356] "Generic (PLEG): container finished" podID="357e99ba-4145-479f-aa06-cf148c2c52b4" containerID="4649efa82104f0303deeedb98930ecf6645f625e240e42b5724dee27b0dcc834" exitCode=0
Oct 30 23:51:20.884640 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:20.884563    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q" event={"ID":"357e99ba-4145-479f-aa06-cf148c2c52b4","Type":"ContainerDied","Data":"4649efa82104f0303deeedb98930ecf6645f625e240e42b5724dee27b0dcc834"}
Oct 30 23:51:20.885066 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:20.885049    5078 scope.go:122] "RemoveContainer" containerID="4649efa82104f0303deeedb98930ecf6645f625e240e42b5724dee27b0dcc834"
Oct 30 23:51:20.886724 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:20.886699    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:51:20.886820 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:20.886765    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:51:20.886820 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:20.886798    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:51:21.016742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.016718    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm"
Oct 30 23:51:21.082442 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.082386    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de020d7e-ce62-476f-9ce1-1ecd344f05b8-kube-api-access-4dt7t\" (UniqueName: \"kubernetes.io/projected/de020d7e-ce62-476f-9ce1-1ecd344f05b8-kube-api-access-4dt7t\") pod \"de020d7e-ce62-476f-9ce1-1ecd344f05b8\" (UID: \"de020d7e-ce62-476f-9ce1-1ecd344f05b8\") "
Oct 30 23:51:21.082442 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.082431    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de020d7e-ce62-476f-9ce1-1ecd344f05b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de020d7e-ce62-476f-9ce1-1ecd344f05b8-plugins-config-volume\") pod \"de020d7e-ce62-476f-9ce1-1ecd344f05b8\" (UID: \"de020d7e-ce62-476f-9ce1-1ecd344f05b8\") "
Oct 30 23:51:21.084624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.083513    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de020d7e-ce62-476f-9ce1-1ecd344f05b8-plugins-config-volume" pod "de020d7e-ce62-476f-9ce1-1ecd344f05b8" (UID: "de020d7e-ce62-476f-9ce1-1ecd344f05b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:51:21.086394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.086357    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de020d7e-ce62-476f-9ce1-1ecd344f05b8-kube-api-access-4dt7t" pod "de020d7e-ce62-476f-9ce1-1ecd344f05b8" (UID: "de020d7e-ce62-476f-9ce1-1ecd344f05b8"). InnerVolumeSpecName "kube-api-access-4dt7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:51:21.111068 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.110972    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv"
Oct 30 23:51:21.185472 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.183985    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8-plugins-config-volume\") pod \"b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8\" (UID: \"b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8\") "
Oct 30 23:51:21.185472 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.184037    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8-kube-api-access-zbt6g\" (UniqueName: \"kubernetes.io/projected/b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8-kube-api-access-zbt6g\") pod \"b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8\" (UID: \"b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8\") "
Oct 30 23:51:21.185472 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.184629    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dt7t\" (UniqueName: \"kubernetes.io/projected/de020d7e-ce62-476f-9ce1-1ecd344f05b8-kube-api-access-4dt7t\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:51:21.185472 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.184654    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de020d7e-ce62-476f-9ce1-1ecd344f05b8-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:51:21.185472 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.185421    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8-plugins-config-volume" pod "b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8" (UID: "b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:51:21.211342 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.211280    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8-kube-api-access-zbt6g" pod "b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8" (UID: "b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8"). InnerVolumeSpecName "kube-api-access-zbt6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:51:21.285922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.285890    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:51:21.285922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.285920    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbt6g\" (UniqueName: \"kubernetes.io/projected/b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8-kube-api-access-zbt6g\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:51:21.893328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.893233    5078 generic.go:356] "Generic (PLEG): container finished" podID="b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8" containerID="0f1b58d42637429fef957c46f4df3331a96128ffe244912612fbbeece223ba3a" exitCode=0
Oct 30 23:51:21.893328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.893280    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv"
Oct 30 23:51:21.893328 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.893309    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv" event={"ID":"b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8","Type":"ContainerDied","Data":"0f1b58d42637429fef957c46f4df3331a96128ffe244912612fbbeece223ba3a"}
Oct 30 23:51:21.893965 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.893359    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv" event={"ID":"b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8","Type":"ContainerDied","Data":"0abb42fb8a0ce941df706958b669f4a7436aaa8fcd31ba046eefd0c15c1f928d"}
Oct 30 23:51:21.893965 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.893382    5078 scope.go:122] "RemoveContainer" containerID="0f1b58d42637429fef957c46f4df3331a96128ffe244912612fbbeece223ba3a"
Oct 30 23:51:21.895582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.895560    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q" event={"ID":"357e99ba-4145-479f-aa06-cf148c2c52b4","Type":"ContainerStarted","Data":"a5601e17240d9a3e10c527feece9248f4ed46ed31f13887b782274298471a93d"}
Oct 30 23:51:21.897261 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.897239    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm" event={"ID":"de020d7e-ce62-476f-9ce1-1ecd344f05b8","Type":"ContainerDied","Data":"64215e86c5606cfb849015197fe862720ad9907d7fb9513770563dc4c027f56e"}
Oct 30 23:51:21.897360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.897271    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm"
Oct 30 23:51:21.904565 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.904545    5078 scope.go:122] "RemoveContainer" containerID="0f1b58d42637429fef957c46f4df3331a96128ffe244912612fbbeece223ba3a"
Oct 30 23:51:21.904881 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:51:21.904859    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f1b58d42637429fef957c46f4df3331a96128ffe244912612fbbeece223ba3a\": container with ID starting with 0f1b58d42637429fef957c46f4df3331a96128ffe244912612fbbeece223ba3a not found: ID does not exist" containerID="0f1b58d42637429fef957c46f4df3331a96128ffe244912612fbbeece223ba3a"
Oct 30 23:51:21.904984 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.904887    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f1b58d42637429fef957c46f4df3331a96128ffe244912612fbbeece223ba3a"} err="failed to get container status \"0f1b58d42637429fef957c46f4df3331a96128ffe244912612fbbeece223ba3a\": rpc error: code = NotFound desc = could not find container \"0f1b58d42637429fef957c46f4df3331a96128ffe244912612fbbeece223ba3a\": container with ID starting with 0f1b58d42637429fef957c46f4df3331a96128ffe244912612fbbeece223ba3a not found: ID does not exist"
Oct 30 23:51:21.904984 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:21.904904    5078 scope.go:122] "RemoveContainer" containerID="7cd2dd1e56af7184ef60d8d45fe5816f29872033898cb17051e00a3b31465050"
Oct 30 23:51:22.185667 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:22.185560    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv"]
Oct 30 23:51:22.188225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:22.188194    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7c36d6-5f9b9cf9cf-7m7rv"]
Oct 30 23:51:22.300899 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:22.300540    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8" path="/var/lib/kubelet/pods/b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8/volumes"
Oct 30 23:51:22.300899 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:22.300863    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm"]
Oct 30 23:51:22.301968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:22.301950    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7c36d6-7fb8485fb7-qwwtm"]
Oct 30 23:51:22.695004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:22.694969    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:51:22.695932 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:22.695906    5078 patch_prober.go:29] interesting pod/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 23:51:22.696033 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:22.695964    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q" podUID="357e99ba-4145-479f-aa06-cf148c2c52b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 23:51:23.694438 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:23.694395    5078 patch_prober.go:29] interesting pod/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 23:51:23.694888 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:23.694453    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q" podUID="357e99ba-4145-479f-aa06-cf148c2c52b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 23:51:24.299052 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:24.299019    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de020d7e-ce62-476f-9ce1-1ecd344f05b8" path="/var/lib/kubelet/pods/de020d7e-ce62-476f-9ce1-1ecd344f05b8/volumes"
Oct 30 23:51:24.694670 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:24.694571    5078 patch_prober.go:29] interesting pod/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 23:51:24.694670 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:24.694661    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q" podUID="357e99ba-4145-479f-aa06-cf148c2c52b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 23:51:24.980677 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:24.975703    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:51:25.694311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:25.694276    5078 patch_prober.go:29] interesting pod/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 23:51:25.694501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:25.694331    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q" podUID="357e99ba-4145-479f-aa06-cf148c2c52b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 23:51:26.694941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:26.694902    5078 patch_prober.go:29] interesting pod/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body=
Oct 30 23:51:26.695373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:26.694956    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q" podUID="357e99ba-4145-479f-aa06-cf148c2c52b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused"
Oct 30 23:51:27.698582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:27.698552    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:51:27.699502 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:27.699484    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:51:38.728641 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:38.728575    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"]
Oct 30 23:51:38.729154 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:38.728825    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c" podUID="412494b5-417a-488e-9ce4-9771d6458325" containerName="istio-proxy" containerID="cri-o://3ffc8952f288a4a2609e8f2f1da8db78733580dd3362178644a5b447184395fe" gracePeriod=30
Oct 30 23:51:43.996860 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:43.996806    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:51:44.076942 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.076910    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-workload-socket\") pod \"412494b5-417a-488e-9ce4-9771d6458325\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") "
Oct 30 23:51:44.077092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.076976    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-credential-socket\") pod \"412494b5-417a-488e-9ce4-9771d6458325\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") "
Oct 30 23:51:44.077092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.077051    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/412494b5-417a-488e-9ce4-9771d6458325-istio-token\" (UniqueName: \"kubernetes.io/projected/412494b5-417a-488e-9ce4-9771d6458325-istio-token\") pod \"412494b5-417a-488e-9ce4-9771d6458325\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") "
Oct 30 23:51:44.077092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.077079    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/412494b5-417a-488e-9ce4-9771d6458325-kube-api-access-6ms5b\" (UniqueName: \"kubernetes.io/projected/412494b5-417a-488e-9ce4-9771d6458325-kube-api-access-6ms5b\") pod \"412494b5-417a-488e-9ce4-9771d6458325\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") "
Oct 30 23:51:44.077260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.077107    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/412494b5-417a-488e-9ce4-9771d6458325-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412494b5-417a-488e-9ce4-9771d6458325-istiod-ca-cert\") pod \"412494b5-417a-488e-9ce4-9771d6458325\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") "
Oct 30 23:51:44.077260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.077163    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-workload-socket" pod "412494b5-417a-488e-9ce4-9771d6458325" (UID: "412494b5-417a-488e-9ce4-9771d6458325"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:51:44.077260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.077185    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-istio-envoy\") pod \"412494b5-417a-488e-9ce4-9771d6458325\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") "
Oct 30 23:51:44.077260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.077221    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-workload-certs\") pod \"412494b5-417a-488e-9ce4-9771d6458325\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") "
Oct 30 23:51:44.077534 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.077289    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-credential-socket" pod "412494b5-417a-488e-9ce4-9771d6458325" (UID: "412494b5-417a-488e-9ce4-9771d6458325"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:51:44.077534 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.077297    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-istio-data\") pod \"412494b5-417a-488e-9ce4-9771d6458325\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") "
Oct 30 23:51:44.077534 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.077394    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/412494b5-417a-488e-9ce4-9771d6458325-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412494b5-417a-488e-9ce4-9771d6458325-istio-podinfo\") pod \"412494b5-417a-488e-9ce4-9771d6458325\" (UID: \"412494b5-417a-488e-9ce4-9771d6458325\") "
Oct 30 23:51:44.077746 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.077590    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/412494b5-417a-488e-9ce4-9771d6458325-istiod-ca-cert" pod "412494b5-417a-488e-9ce4-9771d6458325" (UID: "412494b5-417a-488e-9ce4-9771d6458325"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:51:44.077746 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.077679    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-istio-data" pod "412494b5-417a-488e-9ce4-9771d6458325" (UID: "412494b5-417a-488e-9ce4-9771d6458325"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:51:44.077886 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.077858    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-workload-certs" pod "412494b5-417a-488e-9ce4-9771d6458325" (UID: "412494b5-417a-488e-9ce4-9771d6458325"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:51:44.078184 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.078150    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:51:44.078184 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.078178    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:51:44.078501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.078195    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412494b5-417a-488e-9ce4-9771d6458325-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:51:44.078501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.078209    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:51:44.078501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.078224    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:51:44.079463 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.079431    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-istio-envoy" pod "412494b5-417a-488e-9ce4-9771d6458325" (UID: "412494b5-417a-488e-9ce4-9771d6458325"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:51:44.079583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.079516    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412494b5-417a-488e-9ce4-9771d6458325-kube-api-access-6ms5b" pod "412494b5-417a-488e-9ce4-9771d6458325" (UID: "412494b5-417a-488e-9ce4-9771d6458325"). InnerVolumeSpecName "kube-api-access-6ms5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:51:44.079583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.079536    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412494b5-417a-488e-9ce4-9771d6458325-istio-token" pod "412494b5-417a-488e-9ce4-9771d6458325" (UID: "412494b5-417a-488e-9ce4-9771d6458325"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:51:44.079781 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.079762    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/412494b5-417a-488e-9ce4-9771d6458325-istio-podinfo" pod "412494b5-417a-488e-9ce4-9771d6458325" (UID: "412494b5-417a-488e-9ce4-9771d6458325"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:51:44.088924 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.088897    5078 generic.go:356] "Generic (PLEG): container finished" podID="412494b5-417a-488e-9ce4-9771d6458325" containerID="3ffc8952f288a4a2609e8f2f1da8db78733580dd3362178644a5b447184395fe" exitCode=0
Oct 30 23:51:44.089030 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.088942    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c" event={"ID":"412494b5-417a-488e-9ce4-9771d6458325","Type":"ContainerDied","Data":"3ffc8952f288a4a2609e8f2f1da8db78733580dd3362178644a5b447184395fe"}
Oct 30 23:51:44.089030 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.088980    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c" event={"ID":"412494b5-417a-488e-9ce4-9771d6458325","Type":"ContainerDied","Data":"25324bb0854f554e48ba1f0d4ef0cd5361875c97bc621a6e307d692c7b732d6b"}
Oct 30 23:51:44.089030 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.089006    5078 scope.go:122] "RemoveContainer" containerID="3ffc8952f288a4a2609e8f2f1da8db78733580dd3362178644a5b447184395fe"
Oct 30 23:51:44.089207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.089133    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"
Oct 30 23:51:44.104281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.104262    5078 scope.go:122] "RemoveContainer" containerID="347067e02bdf480b23781cb99d09e82cd306cecd5979478f5d61d6abc399addb"
Oct 30 23:51:44.113355 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.113338    5078 scope.go:122] "RemoveContainer" containerID="3ffc8952f288a4a2609e8f2f1da8db78733580dd3362178644a5b447184395fe"
Oct 30 23:51:44.113639 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:51:44.113621    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ffc8952f288a4a2609e8f2f1da8db78733580dd3362178644a5b447184395fe\": container with ID starting with 3ffc8952f288a4a2609e8f2f1da8db78733580dd3362178644a5b447184395fe not found: ID does not exist" containerID="3ffc8952f288a4a2609e8f2f1da8db78733580dd3362178644a5b447184395fe"
Oct 30 23:51:44.113682 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.113648    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ffc8952f288a4a2609e8f2f1da8db78733580dd3362178644a5b447184395fe"} err="failed to get container status \"3ffc8952f288a4a2609e8f2f1da8db78733580dd3362178644a5b447184395fe\": rpc error: code = NotFound desc = could not find container \"3ffc8952f288a4a2609e8f2f1da8db78733580dd3362178644a5b447184395fe\": container with ID starting with 3ffc8952f288a4a2609e8f2f1da8db78733580dd3362178644a5b447184395fe not found: ID does not exist"
Oct 30 23:51:44.113682 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.113665    5078 scope.go:122] "RemoveContainer" containerID="347067e02bdf480b23781cb99d09e82cd306cecd5979478f5d61d6abc399addb"
Oct 30 23:51:44.113930 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:51:44.113913    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"347067e02bdf480b23781cb99d09e82cd306cecd5979478f5d61d6abc399addb\": container with ID starting with 347067e02bdf480b23781cb99d09e82cd306cecd5979478f5d61d6abc399addb not found: ID does not exist" containerID="347067e02bdf480b23781cb99d09e82cd306cecd5979478f5d61d6abc399addb"
Oct 30 23:51:44.113978 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.113936    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"347067e02bdf480b23781cb99d09e82cd306cecd5979478f5d61d6abc399addb"} err="failed to get container status \"347067e02bdf480b23781cb99d09e82cd306cecd5979478f5d61d6abc399addb\": rpc error: code = NotFound desc = could not find container \"347067e02bdf480b23781cb99d09e82cd306cecd5979478f5d61d6abc399addb\": container with ID starting with 347067e02bdf480b23781cb99d09e82cd306cecd5979478f5d61d6abc399addb not found: ID does not exist"
Oct 30 23:51:44.180143 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.180113    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412494b5-417a-488e-9ce4-9771d6458325-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:51:44.180280 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.180147    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/412494b5-417a-488e-9ce4-9771d6458325-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:51:44.180280 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.180163    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ms5b\" (UniqueName: \"kubernetes.io/projected/412494b5-417a-488e-9ce4-9771d6458325-kube-api-access-6ms5b\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:51:44.180280 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.180180    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412494b5-417a-488e-9ce4-9771d6458325-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:51:44.287122 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.287034    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"]
Oct 30 23:51:44.421624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:44.400173    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-k8d6c"]
Oct 30 23:51:46.299465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:46.299434    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="412494b5-417a-488e-9ce4-9771d6458325" path="/var/lib/kubelet/pods/412494b5-417a-488e-9ce4-9771d6458325/volumes"
Oct 30 23:51:47.593216 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:51:47.593180    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:51:53.107966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.107925    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"]
Oct 30 23:51:53.108470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.108453    5078 cpu_manager.go:414] "containerMap: removing container" podUID="412494b5-417a-488e-9ce4-9771d6458325" containerName="istio-proxy"
Oct 30 23:51:53.108519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.108473    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="412494b5-417a-488e-9ce4-9771d6458325" containerName="istio-proxy"
Oct 30 23:51:53.108519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.108485    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8" containerName="payload-pre-processing"
Oct 30 23:51:53.108519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.108491    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8" containerName="payload-pre-processing"
Oct 30 23:51:53.108519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.108507    5078 cpu_manager.go:414] "containerMap: removing container" podUID="412494b5-417a-488e-9ce4-9771d6458325" containerName="istio-proxy"
Oct 30 23:51:53.108519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.108512    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="412494b5-417a-488e-9ce4-9771d6458325" containerName="istio-proxy"
Oct 30 23:51:53.108519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.108519    5078 cpu_manager.go:414] "containerMap: removing container" podUID="867eeb4a-d196-4cde-b5ca-52145828f464" containerName="revoke-keys"
Oct 30 23:51:53.108796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.108525    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="867eeb4a-d196-4cde-b5ca-52145828f464" containerName="revoke-keys"
Oct 30 23:51:53.108796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.108545    5078 cpu_manager.go:414] "containerMap: removing container" podUID="de020d7e-ce62-476f-9ce1-1ecd344f05b8" containerName="payload-processing"
Oct 30 23:51:53.108796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.108550    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de020d7e-ce62-476f-9ce1-1ecd344f05b8" containerName="payload-processing"
Oct 30 23:51:53.108796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.108657    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="412494b5-417a-488e-9ce4-9771d6458325" containerName="istio-proxy"
Oct 30 23:51:53.108796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.108665    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4e4c1cb-c6d5-49c8-b9bc-675ee4597ce8" containerName="payload-pre-processing"
Oct 30 23:51:53.108796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.108672    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="412494b5-417a-488e-9ce4-9771d6458325" containerName="istio-proxy"
Oct 30 23:51:53.108796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.108678    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="867eeb4a-d196-4cde-b5ca-52145828f464" containerName="revoke-keys"
Oct 30 23:51:53.108796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.108685    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="de020d7e-ce62-476f-9ce1-1ecd344f05b8" containerName="payload-processing"
Oct 30 23:51:53.112022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.112007    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.165809 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.165778    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-276bec\"/\"openshift-service-ca.crt\""
Oct 30 23:51:53.193102 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.193070    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-tmp-dir\") pod \"test-model-276bec-kserve-557fc6f65-ns5vg\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") " pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.193263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.193118    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-home\") pod \"test-model-276bec-kserve-557fc6f65-ns5vg\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") " pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.193263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.193227    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85d3f232-2aef-4854-9cec-933ceaaa9752-tls-certs\") pod \"test-model-276bec-kserve-557fc6f65-ns5vg\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") " pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.193263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.193257    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-dshm\") pod \"test-model-276bec-kserve-557fc6f65-ns5vg\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") " pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.193397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.193294    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-model-cache\") pod \"test-model-276bec-kserve-557fc6f65-ns5vg\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") " pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.193557 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.193534    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mr48\" (UniqueName: \"kubernetes.io/projected/85d3f232-2aef-4854-9cec-933ceaaa9752-kube-api-access-8mr48\") pod \"test-model-276bec-kserve-557fc6f65-ns5vg\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") " pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.194713 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.194695    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-276bec\"/\"kube-root-ca.crt\""
Oct 30 23:51:53.197558 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.197541    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"]
Oct 30 23:51:53.215096 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.215078    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-276bec\"/\"default-dockercfg-wsqhb\""
Oct 30 23:51:53.215268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.215253    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-276bec\"/\"test-model-276bec-kserve-self-signed-certs\""
Oct 30 23:51:53.294417 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.294386    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85d3f232-2aef-4854-9cec-933ceaaa9752-tls-certs\") pod \"test-model-276bec-kserve-557fc6f65-ns5vg\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") " pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.294657 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.294422    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-dshm\") pod \"test-model-276bec-kserve-557fc6f65-ns5vg\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") " pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.294657 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.294459    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-model-cache\") pod \"test-model-276bec-kserve-557fc6f65-ns5vg\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") " pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.294657 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.294651    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mr48\" (UniqueName: \"kubernetes.io/projected/85d3f232-2aef-4854-9cec-933ceaaa9752-kube-api-access-8mr48\") pod \"test-model-276bec-kserve-557fc6f65-ns5vg\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") " pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.294859 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.294691    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-tmp-dir\") pod \"test-model-276bec-kserve-557fc6f65-ns5vg\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") " pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.294859 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.294724    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-home\") pod \"test-model-276bec-kserve-557fc6f65-ns5vg\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") " pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.294996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.294962    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-model-cache\") pod \"test-model-276bec-kserve-557fc6f65-ns5vg\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") " pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.295058 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.295021    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-tmp-dir\") pod \"test-model-276bec-kserve-557fc6f65-ns5vg\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") " pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.295113 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.295053    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-home\") pod \"test-model-276bec-kserve-557fc6f65-ns5vg\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") " pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.296992 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.296972    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-dshm\") pod \"test-model-276bec-kserve-557fc6f65-ns5vg\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") " pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.297271 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.297253    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85d3f232-2aef-4854-9cec-933ceaaa9752-tls-certs\") pod \"test-model-276bec-kserve-557fc6f65-ns5vg\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") " pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.352275 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.352246    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mr48\" (UniqueName: \"kubernetes.io/projected/85d3f232-2aef-4854-9cec-933ceaaa9752-kube-api-access-8mr48\") pod \"test-model-276bec-kserve-557fc6f65-ns5vg\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") " pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.423698 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.423643    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:51:53.804967 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:53.804932    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"]
Oct 30 23:51:53.805111 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:51:53.805061    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85d3f232_2aef_4854_9cec_933ceaaa9752.slice/crio-bc3951db5debe548ca658f940c5999414527ae6c66135a0592e33175712d885a WatchSource:0}: Error finding container bc3951db5debe548ca658f940c5999414527ae6c66135a0592e33175712d885a: Status 404 returned error can't find the container with id bc3951db5debe548ca658f940c5999414527ae6c66135a0592e33175712d885a
Oct 30 23:51:54.153304 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:54.153209    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg" event={"ID":"85d3f232-2aef-4854-9cec-933ceaaa9752","Type":"ContainerStarted","Data":"4131403a409548145f3da42d634f4106b452cbc8c29518e74c55f02816600b69"}
Oct 30 23:51:54.153304 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:54.153249    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg" event={"ID":"85d3f232-2aef-4854-9cec-933ceaaa9752","Type":"ContainerStarted","Data":"bc3951db5debe548ca658f940c5999414527ae6c66135a0592e33175712d885a"}
Oct 30 23:51:54.269580 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:54.269530    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg" podStartSLOduration=1.269516809 podStartE2EDuration="1.269516809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:51:54.266898419 +0000 UTC m=+2700.154804773" watchObservedRunningTime="2026-10-30 23:51:54.269516809 +0000 UTC m=+2700.157423162"
Oct 30 23:51:54.427384 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:51:54.426679    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:52:00.251343 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.251310    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"]
Oct 30 23:52:00.255492 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.255472    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.309111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.309061    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-333b59\"/\"openshift-service-ca.crt\""
Oct 30 23:52:00.334648 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.334620    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-333b59\"/\"kube-root-ca.crt\""
Oct 30 23:52:00.337530 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.337506    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"]
Oct 30 23:52:00.358306 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.358282    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-333b59\"/\"default-dockercfg-sg9pf\""
Oct 30 23:52:00.358306 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.358293    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-333b59\"/\"test-model-333b59-kserve-self-signed-certs\""
Oct 30 23:52:00.382987 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.382962    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-tmp-dir\") pod \"test-model-333b59-kserve-69887dcb9c-bhhg8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") " pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.383108 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.383013    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97878d0b-1dca-46ce-9405-5062573a6ca8-tls-certs\") pod \"test-model-333b59-kserve-69887dcb9c-bhhg8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") " pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.383202 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.383184    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-dshm\") pod \"test-model-333b59-kserve-69887dcb9c-bhhg8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") " pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.383275 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.383238    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp8ch\" (UniqueName: \"kubernetes.io/projected/97878d0b-1dca-46ce-9405-5062573a6ca8-kube-api-access-vp8ch\") pod \"test-model-333b59-kserve-69887dcb9c-bhhg8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") " pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.383330 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.383315    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-model-cache\") pod \"test-model-333b59-kserve-69887dcb9c-bhhg8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") " pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.383488 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.383471    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-home\") pod \"test-model-333b59-kserve-69887dcb9c-bhhg8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") " pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.484282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.484247    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97878d0b-1dca-46ce-9405-5062573a6ca8-tls-certs\") pod \"test-model-333b59-kserve-69887dcb9c-bhhg8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") " pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.484459 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.484324    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-dshm\") pod \"test-model-333b59-kserve-69887dcb9c-bhhg8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") " pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.484459 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.484353    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp8ch\" (UniqueName: \"kubernetes.io/projected/97878d0b-1dca-46ce-9405-5062573a6ca8-kube-api-access-vp8ch\") pod \"test-model-333b59-kserve-69887dcb9c-bhhg8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") " pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.484459 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.484374    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-model-cache\") pod \"test-model-333b59-kserve-69887dcb9c-bhhg8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") " pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.484607 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.484464    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-home\") pod \"test-model-333b59-kserve-69887dcb9c-bhhg8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") " pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.484607 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.484520    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-tmp-dir\") pod \"test-model-333b59-kserve-69887dcb9c-bhhg8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") " pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.484905 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.484877    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-model-cache\") pod \"test-model-333b59-kserve-69887dcb9c-bhhg8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") " pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.485022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.484930    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-home\") pod \"test-model-333b59-kserve-69887dcb9c-bhhg8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") " pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.485022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.484950    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-tmp-dir\") pod \"test-model-333b59-kserve-69887dcb9c-bhhg8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") " pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.486779 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.486754    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-dshm\") pod \"test-model-333b59-kserve-69887dcb9c-bhhg8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") " pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.487017 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.487001    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97878d0b-1dca-46ce-9405-5062573a6ca8-tls-certs\") pod \"test-model-333b59-kserve-69887dcb9c-bhhg8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") " pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.541591 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.541561    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp8ch\" (UniqueName: \"kubernetes.io/projected/97878d0b-1dca-46ce-9405-5062573a6ca8-kube-api-access-vp8ch\") pod \"test-model-333b59-kserve-69887dcb9c-bhhg8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") " pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.567171 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.567137    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:00.945137 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:00.945108    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"]
Oct 30 23:52:00.946012 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:52:00.945981    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97878d0b_1dca_46ce_9405_5062573a6ca8.slice/crio-c7892a3a5427436c6bd2bb0bc7165d72648f2be0d4b6707f2d59505ed00521cf WatchSource:0}: Error finding container c7892a3a5427436c6bd2bb0bc7165d72648f2be0d4b6707f2d59505ed00521cf: Status 404 returned error can't find the container with id c7892a3a5427436c6bd2bb0bc7165d72648f2be0d4b6707f2d59505ed00521cf
Oct 30 23:52:01.195935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:01.195857    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8" event={"ID":"97878d0b-1dca-46ce-9405-5062573a6ca8","Type":"ContainerStarted","Data":"afed14e617b0351f20df9479f7452391138cde54a254ef6daadd4cc78c699afe"}
Oct 30 23:52:01.195935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:01.195893    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8" event={"ID":"97878d0b-1dca-46ce-9405-5062573a6ca8","Type":"ContainerStarted","Data":"c7892a3a5427436c6bd2bb0bc7165d72648f2be0d4b6707f2d59505ed00521cf"}
Oct 30 23:52:01.311713 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:01.311660    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8" podStartSLOduration=1.311644399 podStartE2EDuration="1.311644399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:52:01.309834957 +0000 UTC m=+2707.197741310" watchObservedRunningTime="2026-10-30 23:52:01.311644399 +0000 UTC m=+2707.199550752"
Oct 30 23:52:01.567748 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:01.567712    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:03.424490 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:03.424290    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:52:03.436488 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:03.436464    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:52:03.579295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:03.579266    5078 scope.go:122] "RemoveContainer" containerID="03c55032ec0b0e56ae8ff5591591972df192d30dfb122bd6215f26d77647329b"
Oct 30 23:52:03.588706 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:03.588687    5078 scope.go:122] "RemoveContainer" containerID="832b10462e75a568538368160dfd4cc397036591b914377f717617e1cdf044e3"
Oct 30 23:52:03.597340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:03.597318    5078 scope.go:122] "RemoveContainer" containerID="22f9f7d86b11123e33bb9335b111165225a4bcad20d35e7455f14ac121aa546d"
Oct 30 23:52:04.225710 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:04.225682    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:52:08.877336 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:08.877306    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"]
Oct 30 23:52:08.877889 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:08.877559    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf" podUID="3680396f-1877-4e2e-8226-3591fb4e00e2" containerName="istio-proxy" containerID="cri-o://bae8a16b8a18a1925a68f6cc84668a36e87db5cd777b2855f80a6a284b0b422f" gracePeriod=30
Oct 30 23:52:10.567587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:10.567548    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:10.580625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:10.580572    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:11.267913 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:11.267880    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:52:14.135724 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.135689    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:52:14.265215 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.265113    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-podinfo\") pod \"3680396f-1877-4e2e-8226-3591fb4e00e2\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") "
Oct 30 23:52:14.265408 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.265226    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3680396f-1877-4e2e-8226-3591fb4e00e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3680396f-1877-4e2e-8226-3591fb4e00e2-istiod-ca-cert\") pod \"3680396f-1877-4e2e-8226-3591fb4e00e2\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") "
Oct 30 23:52:14.265408 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.265308    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-data\") pod \"3680396f-1877-4e2e-8226-3591fb4e00e2\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") "
Oct 30 23:52:14.265408 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.265348    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-workload-socket\") pod \"3680396f-1877-4e2e-8226-3591fb4e00e2\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") "
Oct 30 23:52:14.265578 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.265446    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-token\" (UniqueName: \"kubernetes.io/projected/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-token\") pod \"3680396f-1877-4e2e-8226-3591fb4e00e2\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") "
Oct 30 23:52:14.265578 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.265500    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-workload-certs\") pod \"3680396f-1877-4e2e-8226-3591fb4e00e2\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") "
Oct 30 23:52:14.265578 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.265534    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3680396f-1877-4e2e-8226-3591fb4e00e2-kube-api-access-h5r9c\" (UniqueName: \"kubernetes.io/projected/3680396f-1877-4e2e-8226-3591fb4e00e2-kube-api-access-h5r9c\") pod \"3680396f-1877-4e2e-8226-3591fb4e00e2\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") "
Oct 30 23:52:14.265829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.265589    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-credential-socket\") pod \"3680396f-1877-4e2e-8226-3591fb4e00e2\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") "
Oct 30 23:52:14.265829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.265672    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-envoy\") pod \"3680396f-1877-4e2e-8226-3591fb4e00e2\" (UID: \"3680396f-1877-4e2e-8226-3591fb4e00e2\") "
Oct 30 23:52:14.267017 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.266947    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-workload-socket" pod "3680396f-1877-4e2e-8226-3591fb4e00e2" (UID: "3680396f-1877-4e2e-8226-3591fb4e00e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:52:14.267412 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.267215    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-data" pod "3680396f-1877-4e2e-8226-3591fb4e00e2" (UID: "3680396f-1877-4e2e-8226-3591fb4e00e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:52:14.267412 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.267232    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3680396f-1877-4e2e-8226-3591fb4e00e2-istiod-ca-cert" pod "3680396f-1877-4e2e-8226-3591fb4e00e2" (UID: "3680396f-1877-4e2e-8226-3591fb4e00e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:52:14.267412 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.267395    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-credential-socket" pod "3680396f-1877-4e2e-8226-3591fb4e00e2" (UID: "3680396f-1877-4e2e-8226-3591fb4e00e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:52:14.267659 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.267540    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-workload-certs" pod "3680396f-1877-4e2e-8226-3591fb4e00e2" (UID: "3680396f-1877-4e2e-8226-3591fb4e00e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:52:14.270690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.268114    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-podinfo" pod "3680396f-1877-4e2e-8226-3591fb4e00e2" (UID: "3680396f-1877-4e2e-8226-3591fb4e00e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:52:14.270690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.270649    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-token" pod "3680396f-1877-4e2e-8226-3591fb4e00e2" (UID: "3680396f-1877-4e2e-8226-3591fb4e00e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:52:14.272987 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.272961    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-envoy" pod "3680396f-1877-4e2e-8226-3591fb4e00e2" (UID: "3680396f-1877-4e2e-8226-3591fb4e00e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:52:14.273833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.273813    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3680396f-1877-4e2e-8226-3591fb4e00e2-kube-api-access-h5r9c" pod "3680396f-1877-4e2e-8226-3591fb4e00e2" (UID: "3680396f-1877-4e2e-8226-3591fb4e00e2"). InnerVolumeSpecName "kube-api-access-h5r9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:52:14.278714 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.278673    5078 generic.go:356] "Generic (PLEG): container finished" podID="3680396f-1877-4e2e-8226-3591fb4e00e2" containerID="bae8a16b8a18a1925a68f6cc84668a36e87db5cd777b2855f80a6a284b0b422f" exitCode=0
Oct 30 23:52:14.278714 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.278704    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf" event={"ID":"3680396f-1877-4e2e-8226-3591fb4e00e2","Type":"ContainerDied","Data":"bae8a16b8a18a1925a68f6cc84668a36e87db5cd777b2855f80a6a284b0b422f"}
Oct 30 23:52:14.278898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.278738    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"
Oct 30 23:52:14.278898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.278764    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf" event={"ID":"3680396f-1877-4e2e-8226-3591fb4e00e2","Type":"ContainerDied","Data":"3ad69bc151f5c4bd556599d06e263022c160652b8b05b9a71239e261c9c23e8f"}
Oct 30 23:52:14.278898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.278799    5078 scope.go:122] "RemoveContainer" containerID="bae8a16b8a18a1925a68f6cc84668a36e87db5cd777b2855f80a6a284b0b422f"
Oct 30 23:52:14.307049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.307023    5078 scope.go:122] "RemoveContainer" containerID="8e406d786f20b88e30e5f0d6d9a435d1f79a12c927d867b42f4c8c3347659db4"
Oct 30 23:52:14.319025 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.319000    5078 scope.go:122] "RemoveContainer" containerID="bae8a16b8a18a1925a68f6cc84668a36e87db5cd777b2855f80a6a284b0b422f"
Oct 30 23:52:14.319386 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:52:14.319355    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae8a16b8a18a1925a68f6cc84668a36e87db5cd777b2855f80a6a284b0b422f\": container with ID starting with bae8a16b8a18a1925a68f6cc84668a36e87db5cd777b2855f80a6a284b0b422f not found: ID does not exist" containerID="bae8a16b8a18a1925a68f6cc84668a36e87db5cd777b2855f80a6a284b0b422f"
Oct 30 23:52:14.319469 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.319398    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae8a16b8a18a1925a68f6cc84668a36e87db5cd777b2855f80a6a284b0b422f"} err="failed to get container status \"bae8a16b8a18a1925a68f6cc84668a36e87db5cd777b2855f80a6a284b0b422f\": rpc error: code = NotFound desc = could not find container \"bae8a16b8a18a1925a68f6cc84668a36e87db5cd777b2855f80a6a284b0b422f\": container with ID starting with bae8a16b8a18a1925a68f6cc84668a36e87db5cd777b2855f80a6a284b0b422f not found: ID does not exist"
Oct 30 23:52:14.319469 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.319423    5078 scope.go:122] "RemoveContainer" containerID="8e406d786f20b88e30e5f0d6d9a435d1f79a12c927d867b42f4c8c3347659db4"
Oct 30 23:52:14.319768 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:52:14.319748    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e406d786f20b88e30e5f0d6d9a435d1f79a12c927d867b42f4c8c3347659db4\": container with ID starting with 8e406d786f20b88e30e5f0d6d9a435d1f79a12c927d867b42f4c8c3347659db4 not found: ID does not exist" containerID="8e406d786f20b88e30e5f0d6d9a435d1f79a12c927d867b42f4c8c3347659db4"
Oct 30 23:52:14.319837 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.319776    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e406d786f20b88e30e5f0d6d9a435d1f79a12c927d867b42f4c8c3347659db4"} err="failed to get container status \"8e406d786f20b88e30e5f0d6d9a435d1f79a12c927d867b42f4c8c3347659db4\": rpc error: code = NotFound desc = could not find container \"8e406d786f20b88e30e5f0d6d9a435d1f79a12c927d867b42f4c8c3347659db4\": container with ID starting with 8e406d786f20b88e30e5f0d6d9a435d1f79a12c927d867b42f4c8c3347659db4 not found: ID does not exist"
Oct 30 23:52:14.367832 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.367799    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:52:14.367999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.367836    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:52:14.367999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.367852    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:52:14.367999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.367866    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:52:14.367999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.367881    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5r9c\" (UniqueName: \"kubernetes.io/projected/3680396f-1877-4e2e-8226-3591fb4e00e2-kube-api-access-h5r9c\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:52:14.367999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.367899    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:52:14.367999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.367912    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:52:14.367999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.367925    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3680396f-1877-4e2e-8226-3591fb4e00e2-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:52:14.367999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.367938    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3680396f-1877-4e2e-8226-3591fb4e00e2-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:52:14.418222 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.418194    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"]
Oct 30 23:52:14.422992 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:14.422964    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aadaa6-openshift-default-87c69499c-z8mcf"]
Oct 30 23:52:16.297898 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:16.297866    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3680396f-1877-4e2e-8226-3591fb4e00e2" path="/var/lib/kubelet/pods/3680396f-1877-4e2e-8226-3591fb4e00e2/volumes"
Oct 30 23:52:17.591667 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:52:17.591633    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:52:26.046079 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:26.046044    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq"]
Oct 30 23:52:26.095213 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:26.095184    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4e96f80-tqgpq"]
Oct 30 23:52:26.298484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:52:26.298411    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="178c6907-4429-4f22-98c7-99986c942bd8" path="/var/lib/kubelet/pods/178c6907-4429-4f22-98c7-99986c942bd8/volumes"
Oct 30 23:52:47.592961 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:52:47.592927    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:53:07.566064 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:07.566027    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"]
Oct 30 23:53:07.566631 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:07.566230    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx" podUID="4e81b860-0772-48e5-a0a2-8e89095eb245" containerName="istio-proxy" containerID="cri-o://d8637f4c9112212ff8665bc9f8c21f8b885c699afedd95192efd031429c2d743" gracePeriod=30
Oct 30 23:53:07.615375 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:07.615336    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"]
Oct 30 23:53:07.615644 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:07.615584    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59" podUID="6ee01da3-c124-4891-bf71-d6f33f50de56" containerName="istio-proxy" containerID="cri-o://ca2458f8c72a75745c57be4a0d12dd12fa4609ad7bde82037ee762a1e0e236a6" gracePeriod=30
Oct 30 23:53:07.617930 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:07.617907    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"]
Oct 30 23:53:07.618120 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:07.618086    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8" podUID="c8f8595d-81df-4243-bb4c-7afaa5f3ca26" containerName="istio-proxy" containerID="cri-o://fc855c6495858b05ef58491bd002195c0477c5752fe6b3657c1e800cd5a500a2" gracePeriod=30
Oct 30 23:53:07.665573 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:07.665543    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"]
Oct 30 23:53:07.665799 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:07.665776    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77" podUID="f434e905-62f0-4aef-acf5-1ba6f49ff6f0" containerName="istio-proxy" containerID="cri-o://6e23bff946b8ed3e5b31462ce956fe5fca3e025f09dc16829c70672490fe86a4" gracePeriod=30
Oct 30 23:53:08.892316 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:08.892280    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-m5q59 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:53:08.892730 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:08.892358    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59" podUID="6ee01da3-c124-4891-bf71-d6f33f50de56" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:53:12.894727 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.894700    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:53:12.962100 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.962070    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-data\") pod \"4e81b860-0772-48e5-a0a2-8e89095eb245\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") "
Oct 30 23:53:12.962288 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.962146    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-envoy\") pod \"4e81b860-0772-48e5-a0a2-8e89095eb245\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") "
Oct 30 23:53:12.962288 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.962200    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e81b860-0772-48e5-a0a2-8e89095eb245-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e81b860-0772-48e5-a0a2-8e89095eb245-istiod-ca-cert\") pod \"4e81b860-0772-48e5-a0a2-8e89095eb245\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") "
Oct 30 23:53:12.962288 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.962234    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-workload-socket\") pod \"4e81b860-0772-48e5-a0a2-8e89095eb245\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") "
Oct 30 23:53:12.962288 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.962267    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-workload-certs\") pod \"4e81b860-0772-48e5-a0a2-8e89095eb245\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") "
Oct 30 23:53:12.962450 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.962312    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-token\" (UniqueName: \"kubernetes.io/projected/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-token\") pod \"4e81b860-0772-48e5-a0a2-8e89095eb245\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") "
Oct 30 23:53:12.962450 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.962358    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e81b860-0772-48e5-a0a2-8e89095eb245-kube-api-access-f7x47\" (UniqueName: \"kubernetes.io/projected/4e81b860-0772-48e5-a0a2-8e89095eb245-kube-api-access-f7x47\") pod \"4e81b860-0772-48e5-a0a2-8e89095eb245\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") "
Oct 30 23:53:12.962450 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.962386    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-data" pod "4e81b860-0772-48e5-a0a2-8e89095eb245" (UID: "4e81b860-0772-48e5-a0a2-8e89095eb245"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:12.962562 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.962461    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-podinfo\") pod \"4e81b860-0772-48e5-a0a2-8e89095eb245\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") "
Oct 30 23:53:12.962562 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.962492    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-credential-socket\") pod \"4e81b860-0772-48e5-a0a2-8e89095eb245\" (UID: \"4e81b860-0772-48e5-a0a2-8e89095eb245\") "
Oct 30 23:53:12.962653 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.962564    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-workload-socket" pod "4e81b860-0772-48e5-a0a2-8e89095eb245" (UID: "4e81b860-0772-48e5-a0a2-8e89095eb245"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:12.962885 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.962864    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-credential-socket" pod "4e81b860-0772-48e5-a0a2-8e89095eb245" (UID: "4e81b860-0772-48e5-a0a2-8e89095eb245"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:12.963211 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.963186    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e81b860-0772-48e5-a0a2-8e89095eb245-istiod-ca-cert" pod "4e81b860-0772-48e5-a0a2-8e89095eb245" (UID: "4e81b860-0772-48e5-a0a2-8e89095eb245"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:53:12.964046 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.963301    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-workload-certs" pod "4e81b860-0772-48e5-a0a2-8e89095eb245" (UID: "4e81b860-0772-48e5-a0a2-8e89095eb245"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:12.964046 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.963677    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:12.964046 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.963699    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:12.964046 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.963753    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e81b860-0772-48e5-a0a2-8e89095eb245-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:12.964046 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.963894    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:12.964046 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.963908    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:12.966862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.966834    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-envoy" pod "4e81b860-0772-48e5-a0a2-8e89095eb245" (UID: "4e81b860-0772-48e5-a0a2-8e89095eb245"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:12.967102 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.967081    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-podinfo" pod "4e81b860-0772-48e5-a0a2-8e89095eb245" (UID: "4e81b860-0772-48e5-a0a2-8e89095eb245"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:53:12.967258 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.967235    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e81b860-0772-48e5-a0a2-8e89095eb245-kube-api-access-f7x47" pod "4e81b860-0772-48e5-a0a2-8e89095eb245" (UID: "4e81b860-0772-48e5-a0a2-8e89095eb245"). InnerVolumeSpecName "kube-api-access-f7x47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:53:12.967339 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.967321    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-token" pod "4e81b860-0772-48e5-a0a2-8e89095eb245" (UID: "4e81b860-0772-48e5-a0a2-8e89095eb245"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:53:12.995180 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:12.995155    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:53:13.005997 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.005958    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:53:13.011127 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.011099    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:53:13.065271 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.065243    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-podinfo\") pod \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") "
Oct 30 23:53:13.065439 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.065316    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee01da3-c124-4891-bf71-d6f33f50de56-kube-api-access-tw9hl\" (UniqueName: \"kubernetes.io/projected/6ee01da3-c124-4891-bf71-d6f33f50de56-kube-api-access-tw9hl\") pod \"6ee01da3-c124-4891-bf71-d6f33f50de56\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") "
Oct 30 23:53:13.065439 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.065364    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-podinfo\") pod \"6ee01da3-c124-4891-bf71-d6f33f50de56\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") "
Oct 30 23:53:13.065439 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.065413    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-envoy\") pod \"6ee01da3-c124-4891-bf71-d6f33f50de56\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") "
Oct 30 23:53:13.065665 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.065460    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-workload-socket\") pod \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") "
Oct 30 23:53:13.065665 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.065531    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-workload-socket\") pod \"6ee01da3-c124-4891-bf71-d6f33f50de56\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") "
Oct 30 23:53:13.065665 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.065579    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-credential-socket\") pod \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") "
Oct 30 23:53:13.065665 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.065636    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-data\") pod \"6ee01da3-c124-4891-bf71-d6f33f50de56\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") "
Oct 30 23:53:13.065892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.065671    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-envoy\") pod \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") "
Oct 30 23:53:13.065892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.065760    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-data\") pod \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") "
Oct 30 23:53:13.065892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.065791    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ee01da3-c124-4891-bf71-d6f33f50de56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee01da3-c124-4891-bf71-d6f33f50de56-istiod-ca-cert\") pod \"6ee01da3-c124-4891-bf71-d6f33f50de56\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") "
Oct 30 23:53:13.065892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.065833    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-token\" (UniqueName: \"kubernetes.io/projected/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-token\") pod \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") "
Oct 30 23:53:13.066107 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.065899    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-token\" (UniqueName: \"kubernetes.io/projected/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-token\") pod \"6ee01da3-c124-4891-bf71-d6f33f50de56\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") "
Oct 30 23:53:13.066107 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.065933    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-kube-api-access-8v7vp\" (UniqueName: \"kubernetes.io/projected/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-kube-api-access-8v7vp\") pod \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") "
Oct 30 23:53:13.066107 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.065931    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-credential-socket" pod "c8f8595d-81df-4243-bb4c-7afaa5f3ca26" (UID: "c8f8595d-81df-4243-bb4c-7afaa5f3ca26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:13.066107 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.065996    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istiod-ca-cert\") pod \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") "
Oct 30 23:53:13.066107 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.066043    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-credential-socket\") pod \"6ee01da3-c124-4891-bf71-d6f33f50de56\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") "
Oct 30 23:53:13.066107 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.066083    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-workload-certs\") pod \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\" (UID: \"c8f8595d-81df-4243-bb4c-7afaa5f3ca26\") "
Oct 30 23:53:13.066440 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.066118    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-workload-certs\") pod \"6ee01da3-c124-4891-bf71-d6f33f50de56\" (UID: \"6ee01da3-c124-4891-bf71-d6f33f50de56\") "
Oct 30 23:53:13.066440 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.066130    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-workload-socket" pod "6ee01da3-c124-4891-bf71-d6f33f50de56" (UID: "6ee01da3-c124-4891-bf71-d6f33f50de56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:13.066440 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.066137    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-workload-socket" pod "c8f8595d-81df-4243-bb4c-7afaa5f3ca26" (UID: "c8f8595d-81df-4243-bb4c-7afaa5f3ca26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:13.066440 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.066325    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-data" pod "6ee01da3-c124-4891-bf71-d6f33f50de56" (UID: "6ee01da3-c124-4891-bf71-d6f33f50de56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:13.066676 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.066571    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-credential-socket" pod "6ee01da3-c124-4891-bf71-d6f33f50de56" (UID: "6ee01da3-c124-4891-bf71-d6f33f50de56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:13.066836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.066812    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-workload-certs" pod "c8f8595d-81df-4243-bb4c-7afaa5f3ca26" (UID: "c8f8595d-81df-4243-bb4c-7afaa5f3ca26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:13.066907 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.066835    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.066907 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.066859    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.066907 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.066876    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.066907 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.066890    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.066907 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.066905    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.067156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.066920    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.067156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.066934    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.067156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.066949    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7x47\" (UniqueName: \"kubernetes.io/projected/4e81b860-0772-48e5-a0a2-8e89095eb245-kube-api-access-f7x47\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.067156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.066966    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e81b860-0772-48e5-a0a2-8e89095eb245-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.067156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.066971    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-workload-certs" pod "6ee01da3-c124-4891-bf71-d6f33f50de56" (UID: "6ee01da3-c124-4891-bf71-d6f33f50de56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:13.067398 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.067190    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee01da3-c124-4891-bf71-d6f33f50de56-istiod-ca-cert" pod "6ee01da3-c124-4891-bf71-d6f33f50de56" (UID: "6ee01da3-c124-4891-bf71-d6f33f50de56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:53:13.067452 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.067410    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-data" pod "c8f8595d-81df-4243-bb4c-7afaa5f3ca26" (UID: "c8f8595d-81df-4243-bb4c-7afaa5f3ca26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:13.067841 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.067816    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istiod-ca-cert" pod "c8f8595d-81df-4243-bb4c-7afaa5f3ca26" (UID: "c8f8595d-81df-4243-bb4c-7afaa5f3ca26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:53:13.069330 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.069303    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-envoy" pod "c8f8595d-81df-4243-bb4c-7afaa5f3ca26" (UID: "c8f8595d-81df-4243-bb4c-7afaa5f3ca26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:13.069836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.069758    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-kube-api-access-8v7vp" pod "c8f8595d-81df-4243-bb4c-7afaa5f3ca26" (UID: "c8f8595d-81df-4243-bb4c-7afaa5f3ca26"). InnerVolumeSpecName "kube-api-access-8v7vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:53:13.069836 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.069809    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-token" pod "c8f8595d-81df-4243-bb4c-7afaa5f3ca26" (UID: "c8f8595d-81df-4243-bb4c-7afaa5f3ca26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:53:13.070022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.069995    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee01da3-c124-4891-bf71-d6f33f50de56-kube-api-access-tw9hl" pod "6ee01da3-c124-4891-bf71-d6f33f50de56" (UID: "6ee01da3-c124-4891-bf71-d6f33f50de56"). InnerVolumeSpecName "kube-api-access-tw9hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:53:13.070533 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.070509    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-envoy" pod "6ee01da3-c124-4891-bf71-d6f33f50de56" (UID: "6ee01da3-c124-4891-bf71-d6f33f50de56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:13.070700 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.070679    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-token" pod "6ee01da3-c124-4891-bf71-d6f33f50de56" (UID: "6ee01da3-c124-4891-bf71-d6f33f50de56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:53:13.070824 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.070805    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-podinfo" pod "6ee01da3-c124-4891-bf71-d6f33f50de56" (UID: "6ee01da3-c124-4891-bf71-d6f33f50de56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:53:13.071353 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.071336    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-podinfo" pod "c8f8595d-81df-4243-bb4c-7afaa5f3ca26" (UID: "c8f8595d-81df-4243-bb4c-7afaa5f3ca26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:53:13.168406 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.168324    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-credential-socket\") pod \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") "
Oct 30 23:53:13.168554 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.168462    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-envoy\") pod \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") "
Oct 30 23:53:13.168554 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.168507    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-workload-certs\") pod \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") "
Oct 30 23:53:13.168694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.168569    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-podinfo\") pod \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") "
Oct 30 23:53:13.168694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.168626    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istiod-ca-cert\") pod \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") "
Oct 30 23:53:13.168694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.168641    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-credential-socket" pod "f434e905-62f0-4aef-acf5-1ba6f49ff6f0" (UID: "f434e905-62f0-4aef-acf5-1ba6f49ff6f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:13.168879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.168750    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-kube-api-access-9w9pz\" (UniqueName: \"kubernetes.io/projected/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-kube-api-access-9w9pz\") pod \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") "
Oct 30 23:53:13.168879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.168790    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-token\" (UniqueName: \"kubernetes.io/projected/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-token\") pod \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") "
Oct 30 23:53:13.168879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.168868    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-workload-certs" pod "f434e905-62f0-4aef-acf5-1ba6f49ff6f0" (UID: "f434e905-62f0-4aef-acf5-1ba6f49ff6f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:13.169063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.168912    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-data\") pod \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") "
Oct 30 23:53:13.169063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.168945    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istiod-ca-cert" pod "f434e905-62f0-4aef-acf5-1ba6f49ff6f0" (UID: "f434e905-62f0-4aef-acf5-1ba6f49ff6f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:53:13.169063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.168974    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-workload-socket\") pod \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\" (UID: \"f434e905-62f0-4aef-acf5-1ba6f49ff6f0\") "
Oct 30 23:53:13.169249 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.169226    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-data" pod "f434e905-62f0-4aef-acf5-1ba6f49ff6f0" (UID: "f434e905-62f0-4aef-acf5-1ba6f49ff6f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:13.169478 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.169277    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-workload-socket" pod "f434e905-62f0-4aef-acf5-1ba6f49ff6f0" (UID: "f434e905-62f0-4aef-acf5-1ba6f49ff6f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:13.169873 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.169841    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.169873 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.169871    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.170049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.169888    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.170049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.169902    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.170049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.169921    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ee01da3-c124-4891-bf71-d6f33f50de56-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.170049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.169937    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.170049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.169951    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.170049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.169965    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v7vp\" (UniqueName: \"kubernetes.io/projected/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-kube-api-access-8v7vp\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.170049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.169979    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.170049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.169992    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.170049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.170005    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.170049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.170019    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.170049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.170036    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8f8595d-81df-4243-bb4c-7afaa5f3ca26-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.170049 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.170049    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.170793 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.170064    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw9hl\" (UniqueName: \"kubernetes.io/projected/6ee01da3-c124-4891-bf71-d6f33f50de56-kube-api-access-tw9hl\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.170793 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.170078    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.170793 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.170093    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ee01da3-c124-4891-bf71-d6f33f50de56-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.170793 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.170106    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.171050 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.171031    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-kube-api-access-9w9pz" pod "f434e905-62f0-4aef-acf5-1ba6f49ff6f0" (UID: "f434e905-62f0-4aef-acf5-1ba6f49ff6f0"). InnerVolumeSpecName "kube-api-access-9w9pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:53:13.171175 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.171153    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-podinfo" pod "f434e905-62f0-4aef-acf5-1ba6f49ff6f0" (UID: "f434e905-62f0-4aef-acf5-1ba6f49ff6f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:53:13.171391 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.171374    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-envoy" pod "f434e905-62f0-4aef-acf5-1ba6f49ff6f0" (UID: "f434e905-62f0-4aef-acf5-1ba6f49ff6f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:13.171428 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.171395    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-token" pod "f434e905-62f0-4aef-acf5-1ba6f49ff6f0" (UID: "f434e905-62f0-4aef-acf5-1ba6f49ff6f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:53:13.270787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.270751    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.270787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.270779    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w9pz\" (UniqueName: \"kubernetes.io/projected/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-kube-api-access-9w9pz\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.270787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.270790    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.271022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.270798    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f434e905-62f0-4aef-acf5-1ba6f49ff6f0-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:13.669975 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.669947    5078 generic.go:356] "Generic (PLEG): container finished" podID="4e81b860-0772-48e5-a0a2-8e89095eb245" containerID="d8637f4c9112212ff8665bc9f8c21f8b885c699afedd95192efd031429c2d743" exitCode=0
Oct 30 23:53:13.670163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.670017    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"
Oct 30 23:53:13.670163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.670031    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx" event={"ID":"4e81b860-0772-48e5-a0a2-8e89095eb245","Type":"ContainerDied","Data":"d8637f4c9112212ff8665bc9f8c21f8b885c699afedd95192efd031429c2d743"}
Oct 30 23:53:13.670163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.670093    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx" event={"ID":"4e81b860-0772-48e5-a0a2-8e89095eb245","Type":"ContainerDied","Data":"22c13ae39fae1405df501e67122d22bfa5864856c5eafd3dfffb0867bb099483"}
Oct 30 23:53:13.670163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.670122    5078 scope.go:122] "RemoveContainer" containerID="d8637f4c9112212ff8665bc9f8c21f8b885c699afedd95192efd031429c2d743"
Oct 30 23:53:13.671436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.671415    5078 generic.go:356] "Generic (PLEG): container finished" podID="6ee01da3-c124-4891-bf71-d6f33f50de56" containerID="ca2458f8c72a75745c57be4a0d12dd12fa4609ad7bde82037ee762a1e0e236a6" exitCode=0
Oct 30 23:53:13.671582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.671441    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59" event={"ID":"6ee01da3-c124-4891-bf71-d6f33f50de56","Type":"ContainerDied","Data":"ca2458f8c72a75745c57be4a0d12dd12fa4609ad7bde82037ee762a1e0e236a6"}
Oct 30 23:53:13.671582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.671489    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"
Oct 30 23:53:13.671582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.671493    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59" event={"ID":"6ee01da3-c124-4891-bf71-d6f33f50de56","Type":"ContainerDied","Data":"14edee018a1f6e572ed5549f337f5902244e9921a28aae5efd8ab6e4544efa22"}
Oct 30 23:53:13.672990 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.672967    5078 generic.go:356] "Generic (PLEG): container finished" podID="c8f8595d-81df-4243-bb4c-7afaa5f3ca26" containerID="fc855c6495858b05ef58491bd002195c0477c5752fe6b3657c1e800cd5a500a2" exitCode=0
Oct 30 23:53:13.673109 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.672998    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8" event={"ID":"c8f8595d-81df-4243-bb4c-7afaa5f3ca26","Type":"ContainerDied","Data":"fc855c6495858b05ef58491bd002195c0477c5752fe6b3657c1e800cd5a500a2"}
Oct 30 23:53:13.673109 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.673046    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"
Oct 30 23:53:13.673109 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.673087    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8" event={"ID":"c8f8595d-81df-4243-bb4c-7afaa5f3ca26","Type":"ContainerDied","Data":"1744fcaf31222552be557e397177cc6a4a8e2954560b75609659e1fae1da7e35"}
Oct 30 23:53:13.674671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.674650    5078 generic.go:356] "Generic (PLEG): container finished" podID="f434e905-62f0-4aef-acf5-1ba6f49ff6f0" containerID="6e23bff946b8ed3e5b31462ce956fe5fca3e025f09dc16829c70672490fe86a4" exitCode=0
Oct 30 23:53:13.674804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.674714    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"
Oct 30 23:53:13.674804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.674726    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77" event={"ID":"f434e905-62f0-4aef-acf5-1ba6f49ff6f0","Type":"ContainerDied","Data":"6e23bff946b8ed3e5b31462ce956fe5fca3e025f09dc16829c70672490fe86a4"}
Oct 30 23:53:13.674804 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.674761    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77" event={"ID":"f434e905-62f0-4aef-acf5-1ba6f49ff6f0","Type":"ContainerDied","Data":"43598cf89c2e6da6fd00a87e53463d7851df2af406b6b7dfd8d2ef23761c73a9"}
Oct 30 23:53:13.681956 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.681811    5078 scope.go:122] "RemoveContainer" containerID="d8637f4c9112212ff8665bc9f8c21f8b885c699afedd95192efd031429c2d743"
Oct 30 23:53:13.682087 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:53:13.682069    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8637f4c9112212ff8665bc9f8c21f8b885c699afedd95192efd031429c2d743\": container with ID starting with d8637f4c9112212ff8665bc9f8c21f8b885c699afedd95192efd031429c2d743 not found: ID does not exist" containerID="d8637f4c9112212ff8665bc9f8c21f8b885c699afedd95192efd031429c2d743"
Oct 30 23:53:13.682129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.682095    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8637f4c9112212ff8665bc9f8c21f8b885c699afedd95192efd031429c2d743"} err="failed to get container status \"d8637f4c9112212ff8665bc9f8c21f8b885c699afedd95192efd031429c2d743\": rpc error: code = NotFound desc = could not find container \"d8637f4c9112212ff8665bc9f8c21f8b885c699afedd95192efd031429c2d743\": container with ID starting with d8637f4c9112212ff8665bc9f8c21f8b885c699afedd95192efd031429c2d743 not found: ID does not exist"
Oct 30 23:53:13.682129 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.682112    5078 scope.go:122] "RemoveContainer" containerID="ca2458f8c72a75745c57be4a0d12dd12fa4609ad7bde82037ee762a1e0e236a6"
Oct 30 23:53:13.693344 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.693328    5078 scope.go:122] "RemoveContainer" containerID="ca2458f8c72a75745c57be4a0d12dd12fa4609ad7bde82037ee762a1e0e236a6"
Oct 30 23:53:13.693645 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:53:13.693626    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca2458f8c72a75745c57be4a0d12dd12fa4609ad7bde82037ee762a1e0e236a6\": container with ID starting with ca2458f8c72a75745c57be4a0d12dd12fa4609ad7bde82037ee762a1e0e236a6 not found: ID does not exist" containerID="ca2458f8c72a75745c57be4a0d12dd12fa4609ad7bde82037ee762a1e0e236a6"
Oct 30 23:53:13.693716 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.693655    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca2458f8c72a75745c57be4a0d12dd12fa4609ad7bde82037ee762a1e0e236a6"} err="failed to get container status \"ca2458f8c72a75745c57be4a0d12dd12fa4609ad7bde82037ee762a1e0e236a6\": rpc error: code = NotFound desc = could not find container \"ca2458f8c72a75745c57be4a0d12dd12fa4609ad7bde82037ee762a1e0e236a6\": container with ID starting with ca2458f8c72a75745c57be4a0d12dd12fa4609ad7bde82037ee762a1e0e236a6 not found: ID does not exist"
Oct 30 23:53:13.693716 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.693672    5078 scope.go:122] "RemoveContainer" containerID="fc855c6495858b05ef58491bd002195c0477c5752fe6b3657c1e800cd5a500a2"
Oct 30 23:53:13.702614 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.702583    5078 scope.go:122] "RemoveContainer" containerID="fc855c6495858b05ef58491bd002195c0477c5752fe6b3657c1e800cd5a500a2"
Oct 30 23:53:13.702877 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:53:13.702860    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc855c6495858b05ef58491bd002195c0477c5752fe6b3657c1e800cd5a500a2\": container with ID starting with fc855c6495858b05ef58491bd002195c0477c5752fe6b3657c1e800cd5a500a2 not found: ID does not exist" containerID="fc855c6495858b05ef58491bd002195c0477c5752fe6b3657c1e800cd5a500a2"
Oct 30 23:53:13.702923 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.702885    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc855c6495858b05ef58491bd002195c0477c5752fe6b3657c1e800cd5a500a2"} err="failed to get container status \"fc855c6495858b05ef58491bd002195c0477c5752fe6b3657c1e800cd5a500a2\": rpc error: code = NotFound desc = could not find container \"fc855c6495858b05ef58491bd002195c0477c5752fe6b3657c1e800cd5a500a2\": container with ID starting with fc855c6495858b05ef58491bd002195c0477c5752fe6b3657c1e800cd5a500a2 not found: ID does not exist"
Oct 30 23:53:13.702923 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.702900    5078 scope.go:122] "RemoveContainer" containerID="6e23bff946b8ed3e5b31462ce956fe5fca3e025f09dc16829c70672490fe86a4"
Oct 30 23:53:13.711075 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.711061    5078 scope.go:122] "RemoveContainer" containerID="6e23bff946b8ed3e5b31462ce956fe5fca3e025f09dc16829c70672490fe86a4"
Oct 30 23:53:13.711311 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:53:13.711294    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e23bff946b8ed3e5b31462ce956fe5fca3e025f09dc16829c70672490fe86a4\": container with ID starting with 6e23bff946b8ed3e5b31462ce956fe5fca3e025f09dc16829c70672490fe86a4 not found: ID does not exist" containerID="6e23bff946b8ed3e5b31462ce956fe5fca3e025f09dc16829c70672490fe86a4"
Oct 30 23:53:13.711367 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.711321    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e23bff946b8ed3e5b31462ce956fe5fca3e025f09dc16829c70672490fe86a4"} err="failed to get container status \"6e23bff946b8ed3e5b31462ce956fe5fca3e025f09dc16829c70672490fe86a4\": rpc error: code = NotFound desc = could not find container \"6e23bff946b8ed3e5b31462ce956fe5fca3e025f09dc16829c70672490fe86a4\": container with ID starting with 6e23bff946b8ed3e5b31462ce956fe5fca3e025f09dc16829c70672490fe86a4 not found: ID does not exist"
Oct 30 23:53:13.834700 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.834662    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"]
Oct 30 23:53:13.837372 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.837343    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-wzqb8"]
Oct 30 23:53:13.946997 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.946920    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"]
Oct 30 23:53:13.950773 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:13.950751    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-m5q59"]
Oct 30 23:53:14.058520 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:14.058493    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"]
Oct 30 23:53:14.068371 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:14.068346    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-n6bxx"]
Oct 30 23:53:14.175691 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:14.175665    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"]
Oct 30 23:53:14.180446 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:14.180422    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-rwc77"]
Oct 30 23:53:14.302839 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:14.302809    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e81b860-0772-48e5-a0a2-8e89095eb245" path="/var/lib/kubelet/pods/4e81b860-0772-48e5-a0a2-8e89095eb245/volumes"
Oct 30 23:53:14.303297 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:14.303283    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ee01da3-c124-4891-bf71-d6f33f50de56" path="/var/lib/kubelet/pods/6ee01da3-c124-4891-bf71-d6f33f50de56/volumes"
Oct 30 23:53:14.303750 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:14.303736    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8f8595d-81df-4243-bb4c-7afaa5f3ca26" path="/var/lib/kubelet/pods/c8f8595d-81df-4243-bb4c-7afaa5f3ca26/volumes"
Oct 30 23:53:14.304794 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:14.304769    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f434e905-62f0-4aef-acf5-1ba6f49ff6f0" path="/var/lib/kubelet/pods/f434e905-62f0-4aef-acf5-1ba6f49ff6f0/volumes"
Oct 30 23:53:15.050940 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:15.050909    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb"]
Oct 30 23:53:15.100894 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:15.100864    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m"]
Oct 30 23:53:15.102964 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:15.102949    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14c236eb-dxtbb"]
Oct 30 23:53:15.104697 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:15.104679    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd79610e-g929m"]
Oct 30 23:53:16.328905 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:16.325629    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="967439c8-ed26-4000-bacd-809cc04fe909" path="/var/lib/kubelet/pods/967439c8-ed26-4000-bacd-809cc04fe909/volumes"
Oct 30 23:53:16.328905 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:16.326348    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c51425db-16c3-47d4-99f8-1ae1b831772f" path="/var/lib/kubelet/pods/c51425db-16c3-47d4-99f8-1ae1b831772f/volumes"
Oct 30 23:53:17.593134 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:53:17.593101    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:53:18.369589 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:18.369538    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn"]
Oct 30 23:53:18.419102 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:18.419070    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c36d6-z49dn"]
Oct 30 23:53:20.297636 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:20.297588    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="867eeb4a-d196-4cde-b5ca-52145828f464" path="/var/lib/kubelet/pods/867eeb4a-d196-4cde-b5ca-52145828f464/volumes"
Oct 30 23:53:25.245717 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:25.245685    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"]
Oct 30 23:53:25.246108 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:25.245897    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz" podUID="fc1e8c37-0f6c-4a6e-b877-43a235878ca8" containerName="istio-proxy" containerID="cri-o://e8773c79bd0076eb3559abf644ef5e2d49f1a6dc48404bd8a171769d59276a0f" gracePeriod=30
Oct 30 23:53:26.369586 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.369552    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2"]
Oct 30 23:53:26.370093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.370072    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4e81b860-0772-48e5-a0a2-8e89095eb245" containerName="istio-proxy"
Oct 30 23:53:26.370093 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.370090    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e81b860-0772-48e5-a0a2-8e89095eb245" containerName="istio-proxy"
Oct 30 23:53:26.370257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.370105    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f434e905-62f0-4aef-acf5-1ba6f49ff6f0" containerName="istio-proxy"
Oct 30 23:53:26.370257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.370110    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f434e905-62f0-4aef-acf5-1ba6f49ff6f0" containerName="istio-proxy"
Oct 30 23:53:26.370257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.370127    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3680396f-1877-4e2e-8226-3591fb4e00e2" containerName="istio-proxy"
Oct 30 23:53:26.370257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.370132    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3680396f-1877-4e2e-8226-3591fb4e00e2" containerName="istio-proxy"
Oct 30 23:53:26.370257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.370141    5078 cpu_manager.go:414] "containerMap: removing container" podUID="6ee01da3-c124-4891-bf71-d6f33f50de56" containerName="istio-proxy"
Oct 30 23:53:26.370257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.370145    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee01da3-c124-4891-bf71-d6f33f50de56" containerName="istio-proxy"
Oct 30 23:53:26.370257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.370152    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3680396f-1877-4e2e-8226-3591fb4e00e2" containerName="istio-proxy"
Oct 30 23:53:26.370257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.370156    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3680396f-1877-4e2e-8226-3591fb4e00e2" containerName="istio-proxy"
Oct 30 23:53:26.370257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.370166    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c8f8595d-81df-4243-bb4c-7afaa5f3ca26" containerName="istio-proxy"
Oct 30 23:53:26.370257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.370170    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8f8595d-81df-4243-bb4c-7afaa5f3ca26" containerName="istio-proxy"
Oct 30 23:53:26.370257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.370239    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3680396f-1877-4e2e-8226-3591fb4e00e2" containerName="istio-proxy"
Oct 30 23:53:26.370257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.370247    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee01da3-c124-4891-bf71-d6f33f50de56" containerName="istio-proxy"
Oct 30 23:53:26.370257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.370254    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3680396f-1877-4e2e-8226-3591fb4e00e2" containerName="istio-proxy"
Oct 30 23:53:26.370257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.370260    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8f8595d-81df-4243-bb4c-7afaa5f3ca26" containerName="istio-proxy"
Oct 30 23:53:26.370722 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.370268    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e81b860-0772-48e5-a0a2-8e89095eb245" containerName="istio-proxy"
Oct 30 23:53:26.370722 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.370274    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f434e905-62f0-4aef-acf5-1ba6f49ff6f0" containerName="istio-proxy"
Oct 30 23:53:26.373437 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.373422    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2"
Oct 30 23:53:26.429367 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.429341    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2"]
Oct 30 23:53:26.442772 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.442748    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9badfba-c1db-425e-aec7-7a6aa7838d27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2\" (UID: \"c9badfba-c1db-425e-aec7-7a6aa7838d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2"
Oct 30 23:53:26.442904 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.442804    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzvv4\" (UniqueName: \"kubernetes.io/projected/c9badfba-c1db-425e-aec7-7a6aa7838d27-kube-api-access-pzvv4\") pod \"maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2\" (UID: \"c9badfba-c1db-425e-aec7-7a6aa7838d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2"
Oct 30 23:53:26.443031 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.443006    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9badfba-c1db-425e-aec7-7a6aa7838d27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2\" (UID: \"c9badfba-c1db-425e-aec7-7a6aa7838d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2"
Oct 30 23:53:26.475582 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.475560    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:53:26.475742 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.475726    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:53:26.475951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.475938    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:53:26.476007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.475941    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:53:26.544752 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.544722    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9badfba-c1db-425e-aec7-7a6aa7838d27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2\" (UID: \"c9badfba-c1db-425e-aec7-7a6aa7838d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2"
Oct 30 23:53:26.544909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.544829    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9badfba-c1db-425e-aec7-7a6aa7838d27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2\" (UID: \"c9badfba-c1db-425e-aec7-7a6aa7838d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2"
Oct 30 23:53:26.544909 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.544897    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzvv4\" (UniqueName: \"kubernetes.io/projected/c9badfba-c1db-425e-aec7-7a6aa7838d27-kube-api-access-pzvv4\") pod \"maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2\" (UID: \"c9badfba-c1db-425e-aec7-7a6aa7838d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2"
Oct 30 23:53:26.545411 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.545389    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9badfba-c1db-425e-aec7-7a6aa7838d27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2\" (UID: \"c9badfba-c1db-425e-aec7-7a6aa7838d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2"
Oct 30 23:53:26.547403 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.547376    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9badfba-c1db-425e-aec7-7a6aa7838d27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2\" (UID: \"c9badfba-c1db-425e-aec7-7a6aa7838d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2"
Oct 30 23:53:26.604310 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.604287    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzvv4\" (UniqueName: \"kubernetes.io/projected/c9badfba-c1db-425e-aec7-7a6aa7838d27-kube-api-access-pzvv4\") pod \"maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2\" (UID: \"c9badfba-c1db-425e-aec7-7a6aa7838d27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2"
Oct 30 23:53:26.686002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:26.685927    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2"
Oct 30 23:53:27.058234 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:27.058205    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2"]
Oct 30 23:53:27.059713 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:53:27.059683    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9badfba_c1db_425e_aec7_7a6aa7838d27.slice/crio-87ea1ec81e9cb7c468c8cf8127ae419a658934154f73a82b09f3f7039e63f4f0 WatchSource:0}: Error finding container 87ea1ec81e9cb7c468c8cf8127ae419a658934154f73a82b09f3f7039e63f4f0: Status 404 returned error can't find the container with id 87ea1ec81e9cb7c468c8cf8127ae419a658934154f73a82b09f3f7039e63f4f0
Oct 30 23:53:27.081651 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:27.081628    5078 patch_prober.go:29] interesting pod/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:53:27.081754 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:27.081688    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz" podUID="fc1e8c37-0f6c-4a6e-b877-43a235878ca8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:53:27.769973 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:27.769888    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2" event={"ID":"c9badfba-c1db-425e-aec7-7a6aa7838d27","Type":"ContainerStarted","Data":"e4c40014ed812177b589863fdc4d892c69ef694c5035aea93d31d8779ba7fb3c"}
Oct 30 23:53:27.769973 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:27.769945    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2" event={"ID":"c9badfba-c1db-425e-aec7-7a6aa7838d27","Type":"ContainerStarted","Data":"87ea1ec81e9cb7c468c8cf8127ae419a658934154f73a82b09f3f7039e63f4f0"}
Oct 30 23:53:27.883902 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:27.883843    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2" podStartSLOduration=1.883816602 podStartE2EDuration="1.883816602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:27.882964581 +0000 UTC m=+2793.770870935" watchObservedRunningTime="2026-10-30 23:53:27.883816602 +0000 UTC m=+2793.771722955"
Oct 30 23:53:28.777107 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:28.777069    5078 generic.go:356] "Generic (PLEG): container finished" podID="c9badfba-c1db-425e-aec7-7a6aa7838d27" containerID="e4c40014ed812177b589863fdc4d892c69ef694c5035aea93d31d8779ba7fb3c" exitCode=0
Oct 30 23:53:28.777525 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:28.777117    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2" event={"ID":"c9badfba-c1db-425e-aec7-7a6aa7838d27","Type":"ContainerDied","Data":"e4c40014ed812177b589863fdc4d892c69ef694c5035aea93d31d8779ba7fb3c"}
Oct 30 23:53:29.930927 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:29.930899    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2"
Oct 30 23:53:30.010781 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.010752    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9badfba-c1db-425e-aec7-7a6aa7838d27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9badfba-c1db-425e-aec7-7a6aa7838d27-maas-api-openshift-ca\") pod \"c9badfba-c1db-425e-aec7-7a6aa7838d27\" (UID: \"c9badfba-c1db-425e-aec7-7a6aa7838d27\") "
Oct 30 23:53:30.010964 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.010899    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9badfba-c1db-425e-aec7-7a6aa7838d27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9badfba-c1db-425e-aec7-7a6aa7838d27-maas-api-xks-ca\") pod \"c9badfba-c1db-425e-aec7-7a6aa7838d27\" (UID: \"c9badfba-c1db-425e-aec7-7a6aa7838d27\") "
Oct 30 23:53:30.010964 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.010950    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9badfba-c1db-425e-aec7-7a6aa7838d27-kube-api-access-pzvv4\" (UniqueName: \"kubernetes.io/projected/c9badfba-c1db-425e-aec7-7a6aa7838d27-kube-api-access-pzvv4\") pod \"c9badfba-c1db-425e-aec7-7a6aa7838d27\" (UID: \"c9badfba-c1db-425e-aec7-7a6aa7838d27\") "
Oct 30 23:53:30.011133 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.011102    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9badfba-c1db-425e-aec7-7a6aa7838d27-maas-api-openshift-ca" pod "c9badfba-c1db-425e-aec7-7a6aa7838d27" (UID: "c9badfba-c1db-425e-aec7-7a6aa7838d27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:53:30.011426 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.011407    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9badfba-c1db-425e-aec7-7a6aa7838d27-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:30.013121 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.013096    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9badfba-c1db-425e-aec7-7a6aa7838d27-maas-api-xks-ca" pod "c9badfba-c1db-425e-aec7-7a6aa7838d27" (UID: "c9badfba-c1db-425e-aec7-7a6aa7838d27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:53:30.013227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.013208    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9badfba-c1db-425e-aec7-7a6aa7838d27-kube-api-access-pzvv4" pod "c9badfba-c1db-425e-aec7-7a6aa7838d27" (UID: "c9badfba-c1db-425e-aec7-7a6aa7838d27"). InnerVolumeSpecName "kube-api-access-pzvv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:53:30.112128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.112082    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9badfba-c1db-425e-aec7-7a6aa7838d27-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:30.112128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.112124    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzvv4\" (UniqueName: \"kubernetes.io/projected/c9badfba-c1db-425e-aec7-7a6aa7838d27-kube-api-access-pzvv4\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:30.497554 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.497530    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:53:30.617842 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.617766    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-workload-socket\") pod \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") "
Oct 30 23:53:30.618009 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.617854    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-podinfo\") pod \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") "
Oct 30 23:53:30.618009 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.617965    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-data\") pod \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") "
Oct 30 23:53:30.618009 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.618001    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istiod-ca-cert\") pod \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") "
Oct 30 23:53:30.618190 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.618013    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-workload-socket" pod "fc1e8c37-0f6c-4a6e-b877-43a235878ca8" (UID: "fc1e8c37-0f6c-4a6e-b877-43a235878ca8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:30.618190 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.618035    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-kube-api-access-cffmb\" (UniqueName: \"kubernetes.io/projected/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-kube-api-access-cffmb\") pod \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") "
Oct 30 23:53:30.618190 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.618111    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-token\" (UniqueName: \"kubernetes.io/projected/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-token\") pod \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") "
Oct 30 23:53:30.618190 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.618168    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-credential-socket\") pod \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") "
Oct 30 23:53:30.618401 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.618211    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-data" pod "fc1e8c37-0f6c-4a6e-b877-43a235878ca8" (UID: "fc1e8c37-0f6c-4a6e-b877-43a235878ca8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:30.618401 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.618226    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-workload-certs\") pod \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") "
Oct 30 23:53:30.618401 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.618357    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istiod-ca-cert" pod "fc1e8c37-0f6c-4a6e-b877-43a235878ca8" (UID: "fc1e8c37-0f6c-4a6e-b877-43a235878ca8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:53:30.618401 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.618371    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-credential-socket" pod "fc1e8c37-0f6c-4a6e-b877-43a235878ca8" (UID: "fc1e8c37-0f6c-4a6e-b877-43a235878ca8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:30.618656 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.618522    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-workload-certs" pod "fc1e8c37-0f6c-4a6e-b877-43a235878ca8" (UID: "fc1e8c37-0f6c-4a6e-b877-43a235878ca8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:30.618656 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.618558    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-envoy\") pod \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\" (UID: \"fc1e8c37-0f6c-4a6e-b877-43a235878ca8\") "
Oct 30 23:53:30.619034 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.619015    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:30.619131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.619036    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:30.619131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.619050    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:30.619131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.619063    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:30.619131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.619075    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:30.620315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.620286    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-kube-api-access-cffmb" pod "fc1e8c37-0f6c-4a6e-b877-43a235878ca8" (UID: "fc1e8c37-0f6c-4a6e-b877-43a235878ca8"). InnerVolumeSpecName "kube-api-access-cffmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:53:30.620828 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.620808    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-envoy" pod "fc1e8c37-0f6c-4a6e-b877-43a235878ca8" (UID: "fc1e8c37-0f6c-4a6e-b877-43a235878ca8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:30.620828 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.620814    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-podinfo" pod "fc1e8c37-0f6c-4a6e-b877-43a235878ca8" (UID: "fc1e8c37-0f6c-4a6e-b877-43a235878ca8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:53:30.620951 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.620817    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-token" pod "fc1e8c37-0f6c-4a6e-b877-43a235878ca8" (UID: "fc1e8c37-0f6c-4a6e-b877-43a235878ca8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:53:30.719942 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.719913    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:30.719942 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.719938    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cffmb\" (UniqueName: \"kubernetes.io/projected/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-kube-api-access-cffmb\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:30.719942 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.719949    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:30.720164 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.719958    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc1e8c37-0f6c-4a6e-b877-43a235878ca8-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:30.790134 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.790105    5078 generic.go:356] "Generic (PLEG): container finished" podID="fc1e8c37-0f6c-4a6e-b877-43a235878ca8" containerID="e8773c79bd0076eb3559abf644ef5e2d49f1a6dc48404bd8a171769d59276a0f" exitCode=0
Oct 30 23:53:30.790273 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.790150    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"
Oct 30 23:53:30.790273 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.790191    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz" event={"ID":"fc1e8c37-0f6c-4a6e-b877-43a235878ca8","Type":"ContainerDied","Data":"e8773c79bd0076eb3559abf644ef5e2d49f1a6dc48404bd8a171769d59276a0f"}
Oct 30 23:53:30.790273 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.790252    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz" event={"ID":"fc1e8c37-0f6c-4a6e-b877-43a235878ca8","Type":"ContainerDied","Data":"2aaa68bb4e6dda23842df319ce97a22945715fe0e127b5ff1728481763c59037"}
Oct 30 23:53:30.790448 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.790283    5078 scope.go:122] "RemoveContainer" containerID="e8773c79bd0076eb3559abf644ef5e2d49f1a6dc48404bd8a171769d59276a0f"
Oct 30 23:53:30.791863 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.791845    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2"
Oct 30 23:53:30.791968 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.791846    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2" event={"ID":"c9badfba-c1db-425e-aec7-7a6aa7838d27","Type":"ContainerDied","Data":"87ea1ec81e9cb7c468c8cf8127ae419a658934154f73a82b09f3f7039e63f4f0"}
Oct 30 23:53:30.792025 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.791979    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87ea1ec81e9cb7c468c8cf8127ae419a658934154f73a82b09f3f7039e63f4f0"
Oct 30 23:53:30.801840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.801812    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"]
Oct 30 23:53:30.802408 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.802390    5078 scope.go:122] "RemoveContainer" containerID="e76df04832f9ecc5ef738d9f9ce5aaee769e833ab8f4ea1f09235962e2ab58e0"
Oct 30 23:53:30.810105 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.810085    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"]
Oct 30 23:53:30.812978 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.812960    5078 scope.go:122] "RemoveContainer" containerID="e8773c79bd0076eb3559abf644ef5e2d49f1a6dc48404bd8a171769d59276a0f"
Oct 30 23:53:30.813456 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:53:30.813436    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8773c79bd0076eb3559abf644ef5e2d49f1a6dc48404bd8a171769d59276a0f\": container with ID starting with e8773c79bd0076eb3559abf644ef5e2d49f1a6dc48404bd8a171769d59276a0f not found: ID does not exist" containerID="e8773c79bd0076eb3559abf644ef5e2d49f1a6dc48404bd8a171769d59276a0f"
Oct 30 23:53:30.813517 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.813464    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8773c79bd0076eb3559abf644ef5e2d49f1a6dc48404bd8a171769d59276a0f"} err="failed to get container status \"e8773c79bd0076eb3559abf644ef5e2d49f1a6dc48404bd8a171769d59276a0f\": rpc error: code = NotFound desc = could not find container \"e8773c79bd0076eb3559abf644ef5e2d49f1a6dc48404bd8a171769d59276a0f\": container with ID starting with e8773c79bd0076eb3559abf644ef5e2d49f1a6dc48404bd8a171769d59276a0f not found: ID does not exist"
Oct 30 23:53:30.813517 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.813481    5078 scope.go:122] "RemoveContainer" containerID="e76df04832f9ecc5ef738d9f9ce5aaee769e833ab8f4ea1f09235962e2ab58e0"
Oct 30 23:53:30.813816 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:53:30.813796    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e76df04832f9ecc5ef738d9f9ce5aaee769e833ab8f4ea1f09235962e2ab58e0\": container with ID starting with e76df04832f9ecc5ef738d9f9ce5aaee769e833ab8f4ea1f09235962e2ab58e0 not found: ID does not exist" containerID="e76df04832f9ecc5ef738d9f9ce5aaee769e833ab8f4ea1f09235962e2ab58e0"
Oct 30 23:53:30.813884 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:30.813825    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76df04832f9ecc5ef738d9f9ce5aaee769e833ab8f4ea1f09235962e2ab58e0"} err="failed to get container status \"e76df04832f9ecc5ef738d9f9ce5aaee769e833ab8f4ea1f09235962e2ab58e0\": rpc error: code = NotFound desc = could not find container \"e76df04832f9ecc5ef738d9f9ce5aaee769e833ab8f4ea1f09235962e2ab58e0\": container with ID starting with e76df04832f9ecc5ef738d9f9ce5aaee769e833ab8f4ea1f09235962e2ab58e0 not found: ID does not exist"
Oct 30 23:53:31.126115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:31.126070    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"]
Oct 30 23:53:31.129494 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:31.129465    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7c36d6-openshift-default-67d4f6669-8l9cz"]
Oct 30 23:53:32.297201 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:32.297170    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc1e8c37-0f6c-4a6e-b877-43a235878ca8" path="/var/lib/kubelet/pods/fc1e8c37-0f6c-4a6e-b877-43a235878ca8/volumes"
Oct 30 23:53:36.874876 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:36.874799    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g"]
Oct 30 23:53:36.986243 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:36.982183    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t"]
Oct 30 23:53:40.790987 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.790957    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp"]
Oct 30 23:53:40.791389 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.791376    5078 cpu_manager.go:414] "containerMap: removing container" podUID="fc1e8c37-0f6c-4a6e-b877-43a235878ca8" containerName="istio-proxy"
Oct 30 23:53:40.791435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.791390    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1e8c37-0f6c-4a6e-b877-43a235878ca8" containerName="istio-proxy"
Oct 30 23:53:40.791435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.791408    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c9badfba-c1db-425e-aec7-7a6aa7838d27" containerName="revoke-keys"
Oct 30 23:53:40.791435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.791414    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9badfba-c1db-425e-aec7-7a6aa7838d27" containerName="revoke-keys"
Oct 30 23:53:40.791547 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.791505    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1e8c37-0f6c-4a6e-b877-43a235878ca8" containerName="istio-proxy"
Oct 30 23:53:40.791547 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.791514    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9badfba-c1db-425e-aec7-7a6aa7838d27" containerName="revoke-keys"
Oct 30 23:53:40.791547 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.791524    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc1e8c37-0f6c-4a6e-b877-43a235878ca8" containerName="istio-proxy"
Oct 30 23:53:40.796542 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.796525    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp"
Oct 30 23:53:40.851693 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.851667    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp"]
Oct 30 23:53:40.879807 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.879779    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zggxf\" (UniqueName: \"kubernetes.io/projected/87df77dd-cc0e-466b-88de-3397e0ce2fec-kube-api-access-zggxf\") pod \"maas-api-revoke-sub-06cc165c-h7gtp\" (UID: \"87df77dd-cc0e-466b-88de-3397e0ce2fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp"
Oct 30 23:53:40.879944 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.879828    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87df77dd-cc0e-466b-88de-3397e0ce2fec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06cc165c-h7gtp\" (UID: \"87df77dd-cc0e-466b-88de-3397e0ce2fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp"
Oct 30 23:53:40.879988 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.879971    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87df77dd-cc0e-466b-88de-3397e0ce2fec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06cc165c-h7gtp\" (UID: \"87df77dd-cc0e-466b-88de-3397e0ce2fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp"
Oct 30 23:53:40.897815 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.897786    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:53:40.897941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.897841    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:53:40.897941 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.897855    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:53:40.899134 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.899120    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:53:40.981139 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.981116    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zggxf\" (UniqueName: \"kubernetes.io/projected/87df77dd-cc0e-466b-88de-3397e0ce2fec-kube-api-access-zggxf\") pod \"maas-api-revoke-sub-06cc165c-h7gtp\" (UID: \"87df77dd-cc0e-466b-88de-3397e0ce2fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp"
Oct 30 23:53:40.981270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.981156    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87df77dd-cc0e-466b-88de-3397e0ce2fec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06cc165c-h7gtp\" (UID: \"87df77dd-cc0e-466b-88de-3397e0ce2fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp"
Oct 30 23:53:40.981270 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.981250    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87df77dd-cc0e-466b-88de-3397e0ce2fec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06cc165c-h7gtp\" (UID: \"87df77dd-cc0e-466b-88de-3397e0ce2fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp"
Oct 30 23:53:40.981830 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.981811    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87df77dd-cc0e-466b-88de-3397e0ce2fec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06cc165c-h7gtp\" (UID: \"87df77dd-cc0e-466b-88de-3397e0ce2fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp"
Oct 30 23:53:40.983736 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:40.983720    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87df77dd-cc0e-466b-88de-3397e0ce2fec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06cc165c-h7gtp\" (UID: \"87df77dd-cc0e-466b-88de-3397e0ce2fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp"
Oct 30 23:53:41.038069 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:41.038049    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zggxf\" (UniqueName: \"kubernetes.io/projected/87df77dd-cc0e-466b-88de-3397e0ce2fec-kube-api-access-zggxf\") pod \"maas-api-revoke-sub-06cc165c-h7gtp\" (UID: \"87df77dd-cc0e-466b-88de-3397e0ce2fec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp"
Oct 30 23:53:41.109588 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:41.109529    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp"
Oct 30 23:53:41.480545 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:41.480517    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp"]
Oct 30 23:53:41.482254 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:53:41.482222    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87df77dd_cc0e_466b_88de_3397e0ce2fec.slice/crio-22df663077875b8e63a0cd98f6581e987f9632beb79e038b0e40c998ccec5dd9 WatchSource:0}: Error finding container 22df663077875b8e63a0cd98f6581e987f9632beb79e038b0e40c998ccec5dd9: Status 404 returned error can't find the container with id 22df663077875b8e63a0cd98f6581e987f9632beb79e038b0e40c998ccec5dd9
Oct 30 23:53:41.861782 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:41.861724    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp" event={"ID":"87df77dd-cc0e-466b-88de-3397e0ce2fec","Type":"ContainerStarted","Data":"2c4a245ee96b287f58952f8bb52d2c96307a361e718174d437b8987ae3bf0c2d"}
Oct 30 23:53:41.862247 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:41.861834    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp" event={"ID":"87df77dd-cc0e-466b-88de-3397e0ce2fec","Type":"ContainerStarted","Data":"22df663077875b8e63a0cd98f6581e987f9632beb79e038b0e40c998ccec5dd9"}
Oct 30 23:53:41.875161 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:41.875099    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g" podUID="a44f9d90-59b0-4e9d-b95e-e987edcb1266" containerName="payload-processing" containerID="cri-o://3f31a2b4c7ec8e0371dcfca25aa8e0185cb8fb1abadd155dde92374b7ba5b548" gracePeriod=25
Oct 30 23:53:41.982421 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:41.982366    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp" podStartSLOduration=1.982351887 podStartE2EDuration="1.982351887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:41.980172937 +0000 UTC m=+2807.868079287" watchObservedRunningTime="2026-10-30 23:53:41.982351887 +0000 UTC m=+2807.870258236"
Oct 30 23:53:41.987073 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:41.987030    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t" podUID="1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8" containerName="payload-pre-processing" containerID="cri-o://0fa47ece403dbc8d54cade32915e0df883b5f2375993b7e5fd1163eaefa511b4" gracePeriod=25
Oct 30 23:53:42.869361 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:42.869329    5078 generic.go:356] "Generic (PLEG): container finished" podID="87df77dd-cc0e-466b-88de-3397e0ce2fec" containerID="2c4a245ee96b287f58952f8bb52d2c96307a361e718174d437b8987ae3bf0c2d" exitCode=0
Oct 30 23:53:42.869790 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:42.869418    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp" event={"ID":"87df77dd-cc0e-466b-88de-3397e0ce2fec","Type":"ContainerDied","Data":"2c4a245ee96b287f58952f8bb52d2c96307a361e718174d437b8987ae3bf0c2d"}
Oct 30 23:53:44.012987 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:44.012965    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp"
Oct 30 23:53:44.117304 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:44.117268    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87df77dd-cc0e-466b-88de-3397e0ce2fec-kube-api-access-zggxf\" (UniqueName: \"kubernetes.io/projected/87df77dd-cc0e-466b-88de-3397e0ce2fec-kube-api-access-zggxf\") pod \"87df77dd-cc0e-466b-88de-3397e0ce2fec\" (UID: \"87df77dd-cc0e-466b-88de-3397e0ce2fec\") "
Oct 30 23:53:44.117485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:44.117363    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87df77dd-cc0e-466b-88de-3397e0ce2fec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87df77dd-cc0e-466b-88de-3397e0ce2fec-maas-api-xks-ca\") pod \"87df77dd-cc0e-466b-88de-3397e0ce2fec\" (UID: \"87df77dd-cc0e-466b-88de-3397e0ce2fec\") "
Oct 30 23:53:44.117485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:44.117406    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87df77dd-cc0e-466b-88de-3397e0ce2fec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87df77dd-cc0e-466b-88de-3397e0ce2fec-maas-api-openshift-ca\") pod \"87df77dd-cc0e-466b-88de-3397e0ce2fec\" (UID: \"87df77dd-cc0e-466b-88de-3397e0ce2fec\") "
Oct 30 23:53:44.117829 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:44.117805    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87df77dd-cc0e-466b-88de-3397e0ce2fec-maas-api-openshift-ca" pod "87df77dd-cc0e-466b-88de-3397e0ce2fec" (UID: "87df77dd-cc0e-466b-88de-3397e0ce2fec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:53:44.119433 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:44.119416    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87df77dd-cc0e-466b-88de-3397e0ce2fec-kube-api-access-zggxf" pod "87df77dd-cc0e-466b-88de-3397e0ce2fec" (UID: "87df77dd-cc0e-466b-88de-3397e0ce2fec"). InnerVolumeSpecName "kube-api-access-zggxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:53:44.119575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:44.119560    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87df77dd-cc0e-466b-88de-3397e0ce2fec-maas-api-xks-ca" pod "87df77dd-cc0e-466b-88de-3397e0ce2fec" (UID: "87df77dd-cc0e-466b-88de-3397e0ce2fec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:53:44.219305 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:44.219271    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87df77dd-cc0e-466b-88de-3397e0ce2fec-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:44.219305 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:44.219299    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87df77dd-cc0e-466b-88de-3397e0ce2fec-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:44.219305 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:44.219309    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zggxf\" (UniqueName: \"kubernetes.io/projected/87df77dd-cc0e-466b-88de-3397e0ce2fec-kube-api-access-zggxf\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:44.883269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:44.883242    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp"
Oct 30 23:53:44.883269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:44.883254    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp" event={"ID":"87df77dd-cc0e-466b-88de-3397e0ce2fec","Type":"ContainerDied","Data":"22df663077875b8e63a0cd98f6581e987f9632beb79e038b0e40c998ccec5dd9"}
Oct 30 23:53:44.883574 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:44.883301    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22df663077875b8e63a0cd98f6581e987f9632beb79e038b0e40c998ccec5dd9"
Oct 30 23:53:45.823806 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:45.823762    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main" containerID="cri-o://2158d3f2ad787f470e208cdcc290ad48d901ca927108cbf198a94f5b624aaa0a" gracePeriod=45
Oct 30 23:53:45.829827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:45.829793    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main" containerID="cri-o://5813542ea2cb3bb73552cc199db24537fe5bae9445e9c3ad615f7bc2d80cd245" gracePeriod=45
Oct 30 23:53:46.093989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.093911    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:53:46.110097 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.110078    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:53:46.142428 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.142405    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-tmp-dir\") pod \"463812b2-d2a2-49c6-96c2-ee32a4745d71\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") "
Oct 30 23:53:46.142572 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.142478    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22686b57-54a7-4206-9254-65d69fd621db-tls-certs\" (UniqueName: \"kubernetes.io/secret/22686b57-54a7-4206-9254-65d69fd621db-tls-certs\") pod \"22686b57-54a7-4206-9254-65d69fd621db\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") "
Oct 30 23:53:46.142572 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.142496    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-model-cache\") pod \"22686b57-54a7-4206-9254-65d69fd621db\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") "
Oct 30 23:53:46.142572 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.142531    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/463812b2-d2a2-49c6-96c2-ee32a4745d71-kube-api-access-gvql7\" (UniqueName: \"kubernetes.io/projected/463812b2-d2a2-49c6-96c2-ee32a4745d71-kube-api-access-gvql7\") pod \"463812b2-d2a2-49c6-96c2-ee32a4745d71\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") "
Oct 30 23:53:46.142572 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.142566    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/463812b2-d2a2-49c6-96c2-ee32a4745d71-tls-certs\" (UniqueName: \"kubernetes.io/secret/463812b2-d2a2-49c6-96c2-ee32a4745d71-tls-certs\") pod \"463812b2-d2a2-49c6-96c2-ee32a4745d71\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") "
Oct 30 23:53:46.142843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.142650    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-home\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-home\") pod \"22686b57-54a7-4206-9254-65d69fd621db\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") "
Oct 30 23:53:46.142843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.142714    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-tmp-dir" pod "463812b2-d2a2-49c6-96c2-ee32a4745d71" (UID: "463812b2-d2a2-49c6-96c2-ee32a4745d71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:46.142843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.142735    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-dshm\") pod \"22686b57-54a7-4206-9254-65d69fd621db\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") "
Oct 30 23:53:46.143021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.142842    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-model-cache\") pod \"463812b2-d2a2-49c6-96c2-ee32a4745d71\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") "
Oct 30 23:53:46.143021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.142840    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-model-cache" pod "22686b57-54a7-4206-9254-65d69fd621db" (UID: "22686b57-54a7-4206-9254-65d69fd621db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:46.143021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.142874    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-home\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-home\") pod \"463812b2-d2a2-49c6-96c2-ee32a4745d71\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") "
Oct 30 23:53:46.143021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.142899    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-dshm\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-dshm\") pod \"463812b2-d2a2-49c6-96c2-ee32a4745d71\" (UID: \"463812b2-d2a2-49c6-96c2-ee32a4745d71\") "
Oct 30 23:53:46.143021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.142912    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-home" pod "22686b57-54a7-4206-9254-65d69fd621db" (UID: "22686b57-54a7-4206-9254-65d69fd621db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:46.143021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.142947    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22686b57-54a7-4206-9254-65d69fd621db-kube-api-access-fsqgd\" (UniqueName: \"kubernetes.io/projected/22686b57-54a7-4206-9254-65d69fd621db-kube-api-access-fsqgd\") pod \"22686b57-54a7-4206-9254-65d69fd621db\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") "
Oct 30 23:53:46.143021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.142996    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-tmp-dir\") pod \"22686b57-54a7-4206-9254-65d69fd621db\" (UID: \"22686b57-54a7-4206-9254-65d69fd621db\") "
Oct 30 23:53:46.143465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.143114    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-home" pod "463812b2-d2a2-49c6-96c2-ee32a4745d71" (UID: "463812b2-d2a2-49c6-96c2-ee32a4745d71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:46.143465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.143306    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-model-cache" pod "463812b2-d2a2-49c6-96c2-ee32a4745d71" (UID: "463812b2-d2a2-49c6-96c2-ee32a4745d71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:46.143732 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.143504    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-tmp-dir" pod "22686b57-54a7-4206-9254-65d69fd621db" (UID: "22686b57-54a7-4206-9254-65d69fd621db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:46.143732 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.143663    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:46.143732 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.143685    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:46.143732 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.143699    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:46.143732 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.143711    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:46.143732 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.143726    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:46.145226 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.145203    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-dshm" pod "22686b57-54a7-4206-9254-65d69fd621db" (UID: "22686b57-54a7-4206-9254-65d69fd621db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:46.145497 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.145475    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22686b57-54a7-4206-9254-65d69fd621db-tls-certs" pod "22686b57-54a7-4206-9254-65d69fd621db" (UID: "22686b57-54a7-4206-9254-65d69fd621db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:53:46.145640 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.145510    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-dshm" pod "463812b2-d2a2-49c6-96c2-ee32a4745d71" (UID: "463812b2-d2a2-49c6-96c2-ee32a4745d71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:53:46.145640 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.145524    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463812b2-d2a2-49c6-96c2-ee32a4745d71-kube-api-access-gvql7" pod "463812b2-d2a2-49c6-96c2-ee32a4745d71" (UID: "463812b2-d2a2-49c6-96c2-ee32a4745d71"). InnerVolumeSpecName "kube-api-access-gvql7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:53:46.145640 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.145620    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/463812b2-d2a2-49c6-96c2-ee32a4745d71-tls-certs" pod "463812b2-d2a2-49c6-96c2-ee32a4745d71" (UID: "463812b2-d2a2-49c6-96c2-ee32a4745d71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:53:46.145841 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.145824    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22686b57-54a7-4206-9254-65d69fd621db-kube-api-access-fsqgd" pod "22686b57-54a7-4206-9254-65d69fd621db" (UID: "22686b57-54a7-4206-9254-65d69fd621db"). InnerVolumeSpecName "kube-api-access-fsqgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:53:46.245227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.245195    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsqgd\" (UniqueName: \"kubernetes.io/projected/22686b57-54a7-4206-9254-65d69fd621db-kube-api-access-fsqgd\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:46.245227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.245222    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:46.245227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.245231    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/22686b57-54a7-4206-9254-65d69fd621db-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:46.245537 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.245240    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvql7\" (UniqueName: \"kubernetes.io/projected/463812b2-d2a2-49c6-96c2-ee32a4745d71-kube-api-access-gvql7\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:46.245537 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.245251    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/463812b2-d2a2-49c6-96c2-ee32a4745d71-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:46.245537 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.245260    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/22686b57-54a7-4206-9254-65d69fd621db-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:46.245537 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.245268    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/463812b2-d2a2-49c6-96c2-ee32a4745d71-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:46.896721 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.896688    5078 generic.go:356] "Generic (PLEG): container finished" podID="22686b57-54a7-4206-9254-65d69fd621db" containerID="5813542ea2cb3bb73552cc199db24537fe5bae9445e9c3ad615f7bc2d80cd245" exitCode=0
Oct 30 23:53:46.897135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.896738    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"
Oct 30 23:53:46.897135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.896773    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" event={"ID":"22686b57-54a7-4206-9254-65d69fd621db","Type":"ContainerDied","Data":"5813542ea2cb3bb73552cc199db24537fe5bae9445e9c3ad615f7bc2d80cd245"}
Oct 30 23:53:46.897135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.896824    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb" event={"ID":"22686b57-54a7-4206-9254-65d69fd621db","Type":"ContainerDied","Data":"3b4ab65e2cbc7f52daa9a7e761d79d113f0456855856bdda72a510c20d395145"}
Oct 30 23:53:46.897135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.896841    5078 scope.go:122] "RemoveContainer" containerID="5813542ea2cb3bb73552cc199db24537fe5bae9445e9c3ad615f7bc2d80cd245"
Oct 30 23:53:46.898373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.898351    5078 generic.go:356] "Generic (PLEG): container finished" podID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerID="2158d3f2ad787f470e208cdcc290ad48d901ca927108cbf198a94f5b624aaa0a" exitCode=0
Oct 30 23:53:46.898499 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.898396    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"
Oct 30 23:53:46.898499 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.898438    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" event={"ID":"463812b2-d2a2-49c6-96c2-ee32a4745d71","Type":"ContainerDied","Data":"2158d3f2ad787f470e208cdcc290ad48d901ca927108cbf198a94f5b624aaa0a"}
Oct 30 23:53:46.898636 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.898541    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z" event={"ID":"463812b2-d2a2-49c6-96c2-ee32a4745d71","Type":"ContainerDied","Data":"112c499de090d52a4d3e4e451c9bae4b208160a36f5719a6331398ee7d640426"}
Oct 30 23:53:46.907002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.906987    5078 scope.go:122] "RemoveContainer" containerID="5813542ea2cb3bb73552cc199db24537fe5bae9445e9c3ad615f7bc2d80cd245"
Oct 30 23:53:46.907276 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:53:46.907251    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5813542ea2cb3bb73552cc199db24537fe5bae9445e9c3ad615f7bc2d80cd245\": container with ID starting with 5813542ea2cb3bb73552cc199db24537fe5bae9445e9c3ad615f7bc2d80cd245 not found: ID does not exist" containerID="5813542ea2cb3bb73552cc199db24537fe5bae9445e9c3ad615f7bc2d80cd245"
Oct 30 23:53:46.907386 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.907287    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5813542ea2cb3bb73552cc199db24537fe5bae9445e9c3ad615f7bc2d80cd245"} err="failed to get container status \"5813542ea2cb3bb73552cc199db24537fe5bae9445e9c3ad615f7bc2d80cd245\": rpc error: code = NotFound desc = could not find container \"5813542ea2cb3bb73552cc199db24537fe5bae9445e9c3ad615f7bc2d80cd245\": container with ID starting with 5813542ea2cb3bb73552cc199db24537fe5bae9445e9c3ad615f7bc2d80cd245 not found: ID does not exist"
Oct 30 23:53:46.907386 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.907309    5078 scope.go:122] "RemoveContainer" containerID="2158d3f2ad787f470e208cdcc290ad48d901ca927108cbf198a94f5b624aaa0a"
Oct 30 23:53:46.915573 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.915549    5078 scope.go:122] "RemoveContainer" containerID="2158d3f2ad787f470e208cdcc290ad48d901ca927108cbf198a94f5b624aaa0a"
Oct 30 23:53:46.915855 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:53:46.915837    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2158d3f2ad787f470e208cdcc290ad48d901ca927108cbf198a94f5b624aaa0a\": container with ID starting with 2158d3f2ad787f470e208cdcc290ad48d901ca927108cbf198a94f5b624aaa0a not found: ID does not exist" containerID="2158d3f2ad787f470e208cdcc290ad48d901ca927108cbf198a94f5b624aaa0a"
Oct 30 23:53:46.915916 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:46.915863    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2158d3f2ad787f470e208cdcc290ad48d901ca927108cbf198a94f5b624aaa0a"} err="failed to get container status \"2158d3f2ad787f470e208cdcc290ad48d901ca927108cbf198a94f5b624aaa0a\": rpc error: code = NotFound desc = could not find container \"2158d3f2ad787f470e208cdcc290ad48d901ca927108cbf198a94f5b624aaa0a\": container with ID starting with 2158d3f2ad787f470e208cdcc290ad48d901ca927108cbf198a94f5b624aaa0a not found: ID does not exist"
Oct 30 23:53:47.073163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:47.073128    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"]
Oct 30 23:53:47.122426 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:47.122355    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2cfb80/distinct-w3-2cfb80-kserve-74c444c669-mxn9z"]
Oct 30 23:53:47.232157 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:47.232129    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"]
Oct 30 23:53:47.236085 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:47.236060    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-2cfb80/premium-w3-2cfb80-kserve-b889b699d-xcpsb"]
Oct 30 23:53:47.591495 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:53:47.591459    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:53:48.297086 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:48.297055    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22686b57-54a7-4206-9254-65d69fd621db" path="/var/lib/kubelet/pods/22686b57-54a7-4206-9254-65d69fd621db/volumes"
Oct 30 23:53:48.297589 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:48.297574    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" path="/var/lib/kubelet/pods/463812b2-d2a2-49c6-96c2-ee32a4745d71/volumes"
Oct 30 23:53:48.924317 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:48.924279    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g" podUID="a44f9d90-59b0-4e9d-b95e-e987edcb1266" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:53:48.924480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:48.924291    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t" podUID="1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:53:52.744915 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:52.744883    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"]
Oct 30 23:53:53.745068 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.745037    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf"]
Oct 30 23:53:53.745524 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.745509    5078 cpu_manager.go:414] "containerMap: removing container" podUID="87df77dd-cc0e-466b-88de-3397e0ce2fec" containerName="revoke-keys"
Oct 30 23:53:53.745566 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.745526    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87df77dd-cc0e-466b-88de-3397e0ce2fec" containerName="revoke-keys"
Oct 30 23:53:53.745566 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.745551    5078 cpu_manager.go:414] "containerMap: removing container" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main"
Oct 30 23:53:53.745566 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.745556    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main"
Oct 30 23:53:53.745566 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.745565    5078 cpu_manager.go:414] "containerMap: removing container" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main"
Oct 30 23:53:53.745739 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.745570    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main"
Oct 30 23:53:53.745739 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.745578    5078 cpu_manager.go:414] "containerMap: removing container" podUID="fc1e8c37-0f6c-4a6e-b877-43a235878ca8" containerName="istio-proxy"
Oct 30 23:53:53.745739 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.745583    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc1e8c37-0f6c-4a6e-b877-43a235878ca8" containerName="istio-proxy"
Oct 30 23:53:53.745739 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.745685    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="87df77dd-cc0e-466b-88de-3397e0ce2fec" containerName="revoke-keys"
Oct 30 23:53:53.745739 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.745701    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="463812b2-d2a2-49c6-96c2-ee32a4745d71" containerName="main"
Oct 30 23:53:53.745739 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.745710    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="22686b57-54a7-4206-9254-65d69fd621db" containerName="main"
Oct 30 23:53:53.748876 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.748860    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf"
Oct 30 23:53:53.819281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.819242    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:53:53.819444 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.819394    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:53:53.819520 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.819490    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:53:53.821737 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.821711    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf"]
Oct 30 23:53:53.848774 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.848753    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:53:53.856232 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.856208    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh65c\" (UniqueName: \"kubernetes.io/projected/7ad7e150-1c8e-49fb-b952-260caedc2274-kube-api-access-qh65c\") pod \"maas-api-revoke-sub-e4a4e7f9-rxbkf\" (UID: \"7ad7e150-1c8e-49fb-b952-260caedc2274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf"
Oct 30 23:53:53.856386 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.856369    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad7e150-1c8e-49fb-b952-260caedc2274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4a4e7f9-rxbkf\" (UID: \"7ad7e150-1c8e-49fb-b952-260caedc2274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf"
Oct 30 23:53:53.856431 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.856411    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad7e150-1c8e-49fb-b952-260caedc2274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4a4e7f9-rxbkf\" (UID: \"7ad7e150-1c8e-49fb-b952-260caedc2274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf"
Oct 30 23:53:53.957946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.957913    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad7e150-1c8e-49fb-b952-260caedc2274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4a4e7f9-rxbkf\" (UID: \"7ad7e150-1c8e-49fb-b952-260caedc2274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf"
Oct 30 23:53:53.958131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.957960    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad7e150-1c8e-49fb-b952-260caedc2274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4a4e7f9-rxbkf\" (UID: \"7ad7e150-1c8e-49fb-b952-260caedc2274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf"
Oct 30 23:53:53.958131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.958012    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh65c\" (UniqueName: \"kubernetes.io/projected/7ad7e150-1c8e-49fb-b952-260caedc2274-kube-api-access-qh65c\") pod \"maas-api-revoke-sub-e4a4e7f9-rxbkf\" (UID: \"7ad7e150-1c8e-49fb-b952-260caedc2274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf"
Oct 30 23:53:53.958735 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.958713    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad7e150-1c8e-49fb-b952-260caedc2274-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4a4e7f9-rxbkf\" (UID: \"7ad7e150-1c8e-49fb-b952-260caedc2274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf"
Oct 30 23:53:53.960319 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:53.960301    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad7e150-1c8e-49fb-b952-260caedc2274-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4a4e7f9-rxbkf\" (UID: \"7ad7e150-1c8e-49fb-b952-260caedc2274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf"
Oct 30 23:53:54.015472 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:54.015408    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh65c\" (UniqueName: \"kubernetes.io/projected/7ad7e150-1c8e-49fb-b952-260caedc2274-kube-api-access-qh65c\") pod \"maas-api-revoke-sub-e4a4e7f9-rxbkf\" (UID: \"7ad7e150-1c8e-49fb-b952-260caedc2274\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf"
Oct 30 23:53:54.062340 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:54.062312    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf"
Oct 30 23:53:54.432261 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:54.432235    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf"]
Oct 30 23:53:54.433496 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:53:54.433469    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad7e150_1c8e_49fb_b952_260caedc2274.slice/crio-43bd1ed989f39eab8f6b505cd58542ca938f5bc8fb712bde2dba0d8ed3099add WatchSource:0}: Error finding container 43bd1ed989f39eab8f6b505cd58542ca938f5bc8fb712bde2dba0d8ed3099add: Status 404 returned error can't find the container with id 43bd1ed989f39eab8f6b505cd58542ca938f5bc8fb712bde2dba0d8ed3099add
Oct 30 23:53:54.949739 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:54.949705    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf" event={"ID":"7ad7e150-1c8e-49fb-b952-260caedc2274","Type":"ContainerStarted","Data":"60c6f6c530d113a935c5b7cbc14397b46ec352cd661c0e9103b5d747e41ad803"}
Oct 30 23:53:54.950113 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:54.949759    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf" event={"ID":"7ad7e150-1c8e-49fb-b952-260caedc2274","Type":"ContainerStarted","Data":"43bd1ed989f39eab8f6b505cd58542ca938f5bc8fb712bde2dba0d8ed3099add"}
Oct 30 23:53:55.064262 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:55.064215    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf" podStartSLOduration=2.064199928 podStartE2EDuration="2.064199928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:55.062999595 +0000 UTC m=+2820.950905961" watchObservedRunningTime="2026-10-30 23:53:55.064199928 +0000 UTC m=+2820.952106281"
Oct 30 23:53:55.957472 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:55.957435    5078 generic.go:356] "Generic (PLEG): container finished" podID="7ad7e150-1c8e-49fb-b952-260caedc2274" containerID="60c6f6c530d113a935c5b7cbc14397b46ec352cd661c0e9103b5d747e41ad803" exitCode=0
Oct 30 23:53:55.957900 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:55.957496    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf" event={"ID":"7ad7e150-1c8e-49fb-b952-260caedc2274","Type":"ContainerDied","Data":"60c6f6c530d113a935c5b7cbc14397b46ec352cd661c0e9103b5d747e41ad803"}
Oct 30 23:53:57.115193 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:57.115167    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf"
Oct 30 23:53:57.191905 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:57.191869    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ad7e150-1c8e-49fb-b952-260caedc2274-kube-api-access-qh65c\" (UniqueName: \"kubernetes.io/projected/7ad7e150-1c8e-49fb-b952-260caedc2274-kube-api-access-qh65c\") pod \"7ad7e150-1c8e-49fb-b952-260caedc2274\" (UID: \"7ad7e150-1c8e-49fb-b952-260caedc2274\") "
Oct 30 23:53:57.192090 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:57.192031    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ad7e150-1c8e-49fb-b952-260caedc2274-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad7e150-1c8e-49fb-b952-260caedc2274-maas-api-openshift-ca\") pod \"7ad7e150-1c8e-49fb-b952-260caedc2274\" (UID: \"7ad7e150-1c8e-49fb-b952-260caedc2274\") "
Oct 30 23:53:57.192373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:57.192350    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad7e150-1c8e-49fb-b952-260caedc2274-maas-api-openshift-ca" pod "7ad7e150-1c8e-49fb-b952-260caedc2274" (UID: "7ad7e150-1c8e-49fb-b952-260caedc2274"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:53:57.194035 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:57.194011    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad7e150-1c8e-49fb-b952-260caedc2274-kube-api-access-qh65c" pod "7ad7e150-1c8e-49fb-b952-260caedc2274" (UID: "7ad7e150-1c8e-49fb-b952-260caedc2274"). InnerVolumeSpecName "kube-api-access-qh65c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:53:57.293492 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:57.293467    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ad7e150-1c8e-49fb-b952-260caedc2274-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad7e150-1c8e-49fb-b952-260caedc2274-maas-api-xks-ca\") pod \"7ad7e150-1c8e-49fb-b952-260caedc2274\" (UID: \"7ad7e150-1c8e-49fb-b952-260caedc2274\") "
Oct 30 23:53:57.293791 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:57.293776    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh65c\" (UniqueName: \"kubernetes.io/projected/7ad7e150-1c8e-49fb-b952-260caedc2274-kube-api-access-qh65c\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:57.293835 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:57.293795    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ad7e150-1c8e-49fb-b952-260caedc2274-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:57.295653 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:57.295622    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad7e150-1c8e-49fb-b952-260caedc2274-maas-api-xks-ca" pod "7ad7e150-1c8e-49fb-b952-260caedc2274" (UID: "7ad7e150-1c8e-49fb-b952-260caedc2274"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:53:57.394960 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:57.394932    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ad7e150-1c8e-49fb-b952-260caedc2274-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:53:57.970091 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:57.970053    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf" event={"ID":"7ad7e150-1c8e-49fb-b952-260caedc2274","Type":"ContainerDied","Data":"43bd1ed989f39eab8f6b505cd58542ca938f5bc8fb712bde2dba0d8ed3099add"}
Oct 30 23:53:57.970249 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:57.970103    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43bd1ed989f39eab8f6b505cd58542ca938f5bc8fb712bde2dba0d8ed3099add"
Oct 30 23:53:57.970249 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:57.970067    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf"
Oct 30 23:53:58.924468 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:58.924433    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g" podUID="a44f9d90-59b0-4e9d-b95e-e987edcb1266" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:53:58.924891 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:53:58.924469    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t" podUID="1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:54:01.170242 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.170207    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg"]
Oct 30 23:54:01.170714 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.170699    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7ad7e150-1c8e-49fb-b952-260caedc2274" containerName="revoke-keys"
Oct 30 23:54:01.170758 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.170715    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ad7e150-1c8e-49fb-b952-260caedc2274" containerName="revoke-keys"
Oct 30 23:54:01.170803 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.170792    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ad7e150-1c8e-49fb-b952-260caedc2274" containerName="revoke-keys"
Oct 30 23:54:01.174026 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.174009    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg"
Oct 30 23:54:01.278213 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.278182    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:54:01.278370 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.278237    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:54:01.278370 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.278290    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:54:01.278446 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.278399    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:54:01.280931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.280911    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg"]
Oct 30 23:54:01.337389 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.337362    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016b08e4-7659-4907-9ade-5828ec41bc51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-223234f3-jdbvg\" (UID: \"016b08e4-7659-4907-9ade-5828ec41bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg"
Oct 30 23:54:01.337540 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.337400    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqhw8\" (UniqueName: \"kubernetes.io/projected/016b08e4-7659-4907-9ade-5828ec41bc51-kube-api-access-sqhw8\") pod \"maas-api-revoke-sub-223234f3-jdbvg\" (UID: \"016b08e4-7659-4907-9ade-5828ec41bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg"
Oct 30 23:54:01.337685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.337663    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016b08e4-7659-4907-9ade-5828ec41bc51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-223234f3-jdbvg\" (UID: \"016b08e4-7659-4907-9ade-5828ec41bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg"
Oct 30 23:54:01.438961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.438884    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016b08e4-7659-4907-9ade-5828ec41bc51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-223234f3-jdbvg\" (UID: \"016b08e4-7659-4907-9ade-5828ec41bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg"
Oct 30 23:54:01.438961 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.438926    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqhw8\" (UniqueName: \"kubernetes.io/projected/016b08e4-7659-4907-9ade-5828ec41bc51-kube-api-access-sqhw8\") pod \"maas-api-revoke-sub-223234f3-jdbvg\" (UID: \"016b08e4-7659-4907-9ade-5828ec41bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg"
Oct 30 23:54:01.439138 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.439118    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016b08e4-7659-4907-9ade-5828ec41bc51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-223234f3-jdbvg\" (UID: \"016b08e4-7659-4907-9ade-5828ec41bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg"
Oct 30 23:54:01.439791 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.439767    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016b08e4-7659-4907-9ade-5828ec41bc51-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-223234f3-jdbvg\" (UID: \"016b08e4-7659-4907-9ade-5828ec41bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg"
Oct 30 23:54:01.441363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.441341    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016b08e4-7659-4907-9ade-5828ec41bc51-maas-api-xks-ca\") pod \"maas-api-revoke-sub-223234f3-jdbvg\" (UID: \"016b08e4-7659-4907-9ade-5828ec41bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg"
Oct 30 23:54:01.496588 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.496566    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqhw8\" (UniqueName: \"kubernetes.io/projected/016b08e4-7659-4907-9ade-5828ec41bc51-kube-api-access-sqhw8\") pod \"maas-api-revoke-sub-223234f3-jdbvg\" (UID: \"016b08e4-7659-4907-9ade-5828ec41bc51\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg"
Oct 30 23:54:01.662807 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.662776    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn"]
Oct 30 23:54:01.666539 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.666522    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn"
Oct 30 23:54:01.723318 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.723261    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt"]
Oct 30 23:54:01.727115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.727090    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn"]
Oct 30 23:54:01.727225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.727211    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt"
Oct 30 23:54:01.786588 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.786559    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt"]
Oct 30 23:54:01.786760 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.786726    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg"
Oct 30 23:54:01.842827 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.842796    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e4edba-2552-47c2-bf82-69f0f6f2482f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2043c798-rv2jt\" (UID: \"a9e4edba-2552-47c2-bf82-69f0f6f2482f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt"
Oct 30 23:54:01.842984 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.842845    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcl9s\" (UniqueName: \"kubernetes.io/projected/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-kube-api-access-pcl9s\") pod \"maas-api-revoke-sub-cf2bc85f-dhmzn\" (UID: \"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn"
Oct 30 23:54:01.842984 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.842881    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j855d\" (UniqueName: \"kubernetes.io/projected/a9e4edba-2552-47c2-bf82-69f0f6f2482f-kube-api-access-j855d\") pod \"maas-api-revoke-sub-2043c798-rv2jt\" (UID: \"a9e4edba-2552-47c2-bf82-69f0f6f2482f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt"
Oct 30 23:54:01.842984 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.842949    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf2bc85f-dhmzn\" (UID: \"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn"
Oct 30 23:54:01.843167 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.843005    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf2bc85f-dhmzn\" (UID: \"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn"
Oct 30 23:54:01.843167 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.843022    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e4edba-2552-47c2-bf82-69f0f6f2482f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2043c798-rv2jt\" (UID: \"a9e4edba-2552-47c2-bf82-69f0f6f2482f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt"
Oct 30 23:54:01.943650 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.943620    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e4edba-2552-47c2-bf82-69f0f6f2482f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2043c798-rv2jt\" (UID: \"a9e4edba-2552-47c2-bf82-69f0f6f2482f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt"
Oct 30 23:54:01.943758 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.943670    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcl9s\" (UniqueName: \"kubernetes.io/projected/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-kube-api-access-pcl9s\") pod \"maas-api-revoke-sub-cf2bc85f-dhmzn\" (UID: \"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn"
Oct 30 23:54:01.943758 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.943712    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j855d\" (UniqueName: \"kubernetes.io/projected/a9e4edba-2552-47c2-bf82-69f0f6f2482f-kube-api-access-j855d\") pod \"maas-api-revoke-sub-2043c798-rv2jt\" (UID: \"a9e4edba-2552-47c2-bf82-69f0f6f2482f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt"
Oct 30 23:54:01.943876 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.943793    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf2bc85f-dhmzn\" (UID: \"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn"
Oct 30 23:54:01.943876 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.943852    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf2bc85f-dhmzn\" (UID: \"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn"
Oct 30 23:54:01.943991 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.943962    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e4edba-2552-47c2-bf82-69f0f6f2482f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2043c798-rv2jt\" (UID: \"a9e4edba-2552-47c2-bf82-69f0f6f2482f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt"
Oct 30 23:54:01.944644 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.944585    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf2bc85f-dhmzn\" (UID: \"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn"
Oct 30 23:54:01.944764 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.944647    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e4edba-2552-47c2-bf82-69f0f6f2482f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2043c798-rv2jt\" (UID: \"a9e4edba-2552-47c2-bf82-69f0f6f2482f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt"
Oct 30 23:54:01.946088 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.946068    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf2bc85f-dhmzn\" (UID: \"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn"
Oct 30 23:54:01.946189 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.946127    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e4edba-2552-47c2-bf82-69f0f6f2482f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2043c798-rv2jt\" (UID: \"a9e4edba-2552-47c2-bf82-69f0f6f2482f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt"
Oct 30 23:54:01.995423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.995390    5078 generic.go:356] "Generic (PLEG): container finished" podID="a44f9d90-59b0-4e9d-b95e-e987edcb1266" containerID="3f31a2b4c7ec8e0371dcfca25aa8e0185cb8fb1abadd155dde92374b7ba5b548" exitCode=0
Oct 30 23:54:01.995693 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:01.995464    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g" event={"ID":"a44f9d90-59b0-4e9d-b95e-e987edcb1266","Type":"ContainerDied","Data":"3f31a2b4c7ec8e0371dcfca25aa8e0185cb8fb1abadd155dde92374b7ba5b548"}
Oct 30 23:54:02.000996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.000974    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcl9s\" (UniqueName: \"kubernetes.io/projected/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-kube-api-access-pcl9s\") pod \"maas-api-revoke-sub-cf2bc85f-dhmzn\" (UID: \"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn"
Oct 30 23:54:02.001140 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.001118    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j855d\" (UniqueName: \"kubernetes.io/projected/a9e4edba-2552-47c2-bf82-69f0f6f2482f-kube-api-access-j855d\") pod \"maas-api-revoke-sub-2043c798-rv2jt\" (UID: \"a9e4edba-2552-47c2-bf82-69f0f6f2482f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt"
Oct 30 23:54:02.042811 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.042781    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt"
Oct 30 23:54:02.154508 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.154486    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g"
Oct 30 23:54:02.160512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.160482    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg"]
Oct 30 23:54:02.166547 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:54:02.166519    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016b08e4_7659_4907_9ade_5828ec41bc51.slice/crio-98a8cfc86793aa3edc3b5de4ece650f2553218de91213ae553a35d83cce7c292 WatchSource:0}: Error finding container 98a8cfc86793aa3edc3b5de4ece650f2553218de91213ae553a35d83cce7c292: Status 404 returned error can't find the container with id 98a8cfc86793aa3edc3b5de4ece650f2553218de91213ae553a35d83cce7c292
Oct 30 23:54:02.248164 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.248136    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44f9d90-59b0-4e9d-b95e-e987edcb1266-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44f9d90-59b0-4e9d-b95e-e987edcb1266-plugins-config-volume\") pod \"a44f9d90-59b0-4e9d-b95e-e987edcb1266\" (UID: \"a44f9d90-59b0-4e9d-b95e-e987edcb1266\") "
Oct 30 23:54:02.248492 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.248204    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44f9d90-59b0-4e9d-b95e-e987edcb1266-kube-api-access-zjnsb\" (UniqueName: \"kubernetes.io/projected/a44f9d90-59b0-4e9d-b95e-e987edcb1266-kube-api-access-zjnsb\") pod \"a44f9d90-59b0-4e9d-b95e-e987edcb1266\" (UID: \"a44f9d90-59b0-4e9d-b95e-e987edcb1266\") "
Oct 30 23:54:02.248548 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.248507    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44f9d90-59b0-4e9d-b95e-e987edcb1266-plugins-config-volume" pod "a44f9d90-59b0-4e9d-b95e-e987edcb1266" (UID: "a44f9d90-59b0-4e9d-b95e-e987edcb1266"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:54:02.248735 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.248715    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44f9d90-59b0-4e9d-b95e-e987edcb1266-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:02.250358 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.250337    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44f9d90-59b0-4e9d-b95e-e987edcb1266-kube-api-access-zjnsb" pod "a44f9d90-59b0-4e9d-b95e-e987edcb1266" (UID: "a44f9d90-59b0-4e9d-b95e-e987edcb1266"). InnerVolumeSpecName "kube-api-access-zjnsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:54:02.254944 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.254924    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t"
Oct 30 23:54:02.280070 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.280047    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn"
Oct 30 23:54:02.350221 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.350187    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8-plugins-config-volume\") pod \"1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8\" (UID: \"1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8\") "
Oct 30 23:54:02.350335 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.350248    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8-kube-api-access-sb8fr\" (UniqueName: \"kubernetes.io/projected/1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8-kube-api-access-sb8fr\") pod \"1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8\" (UID: \"1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8\") "
Oct 30 23:54:02.350660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.350628    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8-plugins-config-volume" pod "1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8" (UID: "1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:54:02.350936 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.350915    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:02.351017 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.350946    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjnsb\" (UniqueName: \"kubernetes.io/projected/a44f9d90-59b0-4e9d-b95e-e987edcb1266-kube-api-access-zjnsb\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:02.353688 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.353664    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8-kube-api-access-sb8fr" pod "1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8" (UID: "1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8"). InnerVolumeSpecName "kube-api-access-sb8fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:54:02.421141 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.421120    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt"]
Oct 30 23:54:02.422464 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:54:02.422437    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e4edba_2552_47c2_bf82_69f0f6f2482f.slice/crio-2d11a95def1f74481957ab099c5f4bb264c9a045a20a010df1ea9b37ba04a1f5 WatchSource:0}: Error finding container 2d11a95def1f74481957ab099c5f4bb264c9a045a20a010df1ea9b37ba04a1f5: Status 404 returned error can't find the container with id 2d11a95def1f74481957ab099c5f4bb264c9a045a20a010df1ea9b37ba04a1f5
Oct 30 23:54:02.451954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.451812    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb8fr\" (UniqueName: \"kubernetes.io/projected/1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8-kube-api-access-sb8fr\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:02.679521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:02.679496    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn"]
Oct 30 23:54:02.680368 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:54:02.680347    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6bf5bf_83fc_4be2_b5d0_a42a6f821132.slice/crio-d24bace7799eab49406d8523f5574d5f4643b12876397f6396f68b8864024549 WatchSource:0}: Error finding container d24bace7799eab49406d8523f5574d5f4643b12876397f6396f68b8864024549: Status 404 returned error can't find the container with id d24bace7799eab49406d8523f5574d5f4643b12876397f6396f68b8864024549
Oct 30 23:54:03.003257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.003209    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt" event={"ID":"a9e4edba-2552-47c2-bf82-69f0f6f2482f","Type":"ContainerStarted","Data":"c421d31b32e3b376d947231c2c8506e5a07ebd5990815903429ced1e27225156"}
Oct 30 23:54:03.003412 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.003274    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt" event={"ID":"a9e4edba-2552-47c2-bf82-69f0f6f2482f","Type":"ContainerStarted","Data":"2d11a95def1f74481957ab099c5f4bb264c9a045a20a010df1ea9b37ba04a1f5"}
Oct 30 23:54:03.004903 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.004871    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn" event={"ID":"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132","Type":"ContainerStarted","Data":"e58ce3f2c8553035b8ee3d532ad9351a998f25a47cff936a1e81d90356b76702"}
Oct 30 23:54:03.005022 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.004923    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn" event={"ID":"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132","Type":"ContainerStarted","Data":"d24bace7799eab49406d8523f5574d5f4643b12876397f6396f68b8864024549"}
Oct 30 23:54:03.006186 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.006162    5078 generic.go:356] "Generic (PLEG): container finished" podID="1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8" containerID="0fa47ece403dbc8d54cade32915e0df883b5f2375993b7e5fd1163eaefa511b4" exitCode=0
Oct 30 23:54:03.006281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.006203    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t"
Oct 30 23:54:03.006281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.006243    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t" event={"ID":"1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8","Type":"ContainerDied","Data":"0fa47ece403dbc8d54cade32915e0df883b5f2375993b7e5fd1163eaefa511b4"}
Oct 30 23:54:03.006389 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.006288    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t" event={"ID":"1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8","Type":"ContainerDied","Data":"2df545704e245c958b6f3e6a53243694a3cc51df46a2e141a51b81752453c51b"}
Oct 30 23:54:03.006389 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.006312    5078 scope.go:122] "RemoveContainer" containerID="0fa47ece403dbc8d54cade32915e0df883b5f2375993b7e5fd1163eaefa511b4"
Oct 30 23:54:03.008148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.008126    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg" event={"ID":"016b08e4-7659-4907-9ade-5828ec41bc51","Type":"ContainerStarted","Data":"23a439e80f03426effbe94af954805c4a6e99c41543236836cfa885c658b2bc3"}
Oct 30 23:54:03.008267 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.008172    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg" event={"ID":"016b08e4-7659-4907-9ade-5828ec41bc51","Type":"ContainerStarted","Data":"98a8cfc86793aa3edc3b5de4ece650f2553218de91213ae553a35d83cce7c292"}
Oct 30 23:54:03.009664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.009643    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g" event={"ID":"a44f9d90-59b0-4e9d-b95e-e987edcb1266","Type":"ContainerDied","Data":"612cf5731622a6a3840e7c6d4066acada28c7028bea073df1cfaf962482b45eb"}
Oct 30 23:54:03.009664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.009661    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g"
Oct 30 23:54:03.017269 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.017253    5078 scope.go:122] "RemoveContainer" containerID="0fa47ece403dbc8d54cade32915e0df883b5f2375993b7e5fd1163eaefa511b4"
Oct 30 23:54:03.017512 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:54:03.017487    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fa47ece403dbc8d54cade32915e0df883b5f2375993b7e5fd1163eaefa511b4\": container with ID starting with 0fa47ece403dbc8d54cade32915e0df883b5f2375993b7e5fd1163eaefa511b4 not found: ID does not exist" containerID="0fa47ece403dbc8d54cade32915e0df883b5f2375993b7e5fd1163eaefa511b4"
Oct 30 23:54:03.017566 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.017518    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa47ece403dbc8d54cade32915e0df883b5f2375993b7e5fd1163eaefa511b4"} err="failed to get container status \"0fa47ece403dbc8d54cade32915e0df883b5f2375993b7e5fd1163eaefa511b4\": rpc error: code = NotFound desc = could not find container \"0fa47ece403dbc8d54cade32915e0df883b5f2375993b7e5fd1163eaefa511b4\": container with ID starting with 0fa47ece403dbc8d54cade32915e0df883b5f2375993b7e5fd1163eaefa511b4 not found: ID does not exist"
Oct 30 23:54:03.017566 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.017534    5078 scope.go:122] "RemoveContainer" containerID="3f31a2b4c7ec8e0371dcfca25aa8e0185cb8fb1abadd155dde92374b7ba5b548"
Oct 30 23:54:03.128849 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.128766    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt" podStartSLOduration=2.128747848 podStartE2EDuration="2.128747848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:03.125588701 +0000 UTC m=+2829.013495054" watchObservedRunningTime="2026-10-30 23:54:03.128747848 +0000 UTC m=+2829.016654200"
Oct 30 23:54:03.237630 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.237583    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g"]
Oct 30 23:54:03.241720 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.241698    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-2cfb80-774c6bf479-m6t8g"]
Oct 30 23:54:03.353859 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.353807    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg" podStartSLOduration=2.353793531 podStartE2EDuration="2.353793531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:03.350781457 +0000 UTC m=+2829.238687810" watchObservedRunningTime="2026-10-30 23:54:03.353793531 +0000 UTC m=+2829.241699884"
Oct 30 23:54:03.461232 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.461148    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t"]
Oct 30 23:54:03.465354 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.465328    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-2cfb80-558b697f95-kv97t"]
Oct 30 23:54:03.577261 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.577211    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn" podStartSLOduration=2.5771925959999997 podStartE2EDuration="2.577192596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:03.575448579 +0000 UTC m=+2829.463354957" watchObservedRunningTime="2026-10-30 23:54:03.577192596 +0000 UTC m=+2829.465098950"
Oct 30 23:54:03.659908 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.659877    5078 scope.go:122] "RemoveContainer" containerID="7785750cb5febc980da72f883b469889b091aacefa26f131de05725b29ed4a9a"
Oct 30 23:54:03.669235 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:03.669219    5078 scope.go:122] "RemoveContainer" containerID="3d82b30e7194fdc4a9a0d2293ab14f248dc137fd1617360f5912dbf7065e463a"
Oct 30 23:54:04.017202 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:04.017116    5078 generic.go:356] "Generic (PLEG): container finished" podID="a9e4edba-2552-47c2-bf82-69f0f6f2482f" containerID="c421d31b32e3b376d947231c2c8506e5a07ebd5990815903429ced1e27225156" exitCode=0
Oct 30 23:54:04.017373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:04.017205    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt" event={"ID":"a9e4edba-2552-47c2-bf82-69f0f6f2482f","Type":"ContainerDied","Data":"c421d31b32e3b376d947231c2c8506e5a07ebd5990815903429ced1e27225156"}
Oct 30 23:54:04.018683 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:04.018654    5078 generic.go:356] "Generic (PLEG): container finished" podID="1a6bf5bf-83fc-4be2-b5d0-a42a6f821132" containerID="e58ce3f2c8553035b8ee3d532ad9351a998f25a47cff936a1e81d90356b76702" exitCode=0
Oct 30 23:54:04.018807 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:04.018739    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn" event={"ID":"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132","Type":"ContainerDied","Data":"e58ce3f2c8553035b8ee3d532ad9351a998f25a47cff936a1e81d90356b76702"}
Oct 30 23:54:04.020244 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:04.020225    5078 generic.go:356] "Generic (PLEG): container finished" podID="016b08e4-7659-4907-9ade-5828ec41bc51" containerID="23a439e80f03426effbe94af954805c4a6e99c41543236836cfa885c658b2bc3" exitCode=0
Oct 30 23:54:04.020342 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:04.020292    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg" event={"ID":"016b08e4-7659-4907-9ade-5828ec41bc51","Type":"ContainerDied","Data":"23a439e80f03426effbe94af954805c4a6e99c41543236836cfa885c658b2bc3"}
Oct 30 23:54:04.298486 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:04.298445    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8" path="/var/lib/kubelet/pods/1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8/volumes"
Oct 30 23:54:04.298969 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:04.298947    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a44f9d90-59b0-4e9d-b95e-e987edcb1266" path="/var/lib/kubelet/pods/a44f9d90-59b0-4e9d-b95e-e987edcb1266/volumes"
Oct 30 23:54:05.178688 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.177616    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt"
Oct 30 23:54:05.184700 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.184669    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9e4edba-2552-47c2-bf82-69f0f6f2482f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e4edba-2552-47c2-bf82-69f0f6f2482f-maas-api-xks-ca\") pod \"a9e4edba-2552-47c2-bf82-69f0f6f2482f\" (UID: \"a9e4edba-2552-47c2-bf82-69f0f6f2482f\") "
Oct 30 23:54:05.184855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.184742    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9e4edba-2552-47c2-bf82-69f0f6f2482f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e4edba-2552-47c2-bf82-69f0f6f2482f-maas-api-openshift-ca\") pod \"a9e4edba-2552-47c2-bf82-69f0f6f2482f\" (UID: \"a9e4edba-2552-47c2-bf82-69f0f6f2482f\") "
Oct 30 23:54:05.186171 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.185067    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e4edba-2552-47c2-bf82-69f0f6f2482f-kube-api-access-j855d\" (UniqueName: \"kubernetes.io/projected/a9e4edba-2552-47c2-bf82-69f0f6f2482f-kube-api-access-j855d\") pod \"a9e4edba-2552-47c2-bf82-69f0f6f2482f\" (UID: \"a9e4edba-2552-47c2-bf82-69f0f6f2482f\") "
Oct 30 23:54:05.186171 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.185223    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e4edba-2552-47c2-bf82-69f0f6f2482f-maas-api-openshift-ca" pod "a9e4edba-2552-47c2-bf82-69f0f6f2482f" (UID: "a9e4edba-2552-47c2-bf82-69f0f6f2482f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:54:05.186171 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.185581    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e4edba-2552-47c2-bf82-69f0f6f2482f-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:05.187892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.187369    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e4edba-2552-47c2-bf82-69f0f6f2482f-maas-api-xks-ca" pod "a9e4edba-2552-47c2-bf82-69f0f6f2482f" (UID: "a9e4edba-2552-47c2-bf82-69f0f6f2482f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:54:05.188026 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.187938    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e4edba-2552-47c2-bf82-69f0f6f2482f-kube-api-access-j855d" pod "a9e4edba-2552-47c2-bf82-69f0f6f2482f" (UID: "a9e4edba-2552-47c2-bf82-69f0f6f2482f"). InnerVolumeSpecName "kube-api-access-j855d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:54:05.236249 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.236225    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn"
Oct 30 23:54:05.256470 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.256449    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg"
Oct 30 23:54:05.286587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.286529    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-kube-api-access-pcl9s\" (UniqueName: \"kubernetes.io/projected/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-kube-api-access-pcl9s\") pod \"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132\" (UID: \"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132\") "
Oct 30 23:54:05.286716 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.286611    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-maas-api-openshift-ca\") pod \"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132\" (UID: \"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132\") "
Oct 30 23:54:05.286716 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.286652    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-maas-api-xks-ca\") pod \"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132\" (UID: \"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132\") "
Oct 30 23:54:05.286716 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.286698    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/016b08e4-7659-4907-9ade-5828ec41bc51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016b08e4-7659-4907-9ade-5828ec41bc51-maas-api-xks-ca\") pod \"016b08e4-7659-4907-9ade-5828ec41bc51\" (UID: \"016b08e4-7659-4907-9ade-5828ec41bc51\") "
Oct 30 23:54:05.287208 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.286744    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/016b08e4-7659-4907-9ade-5828ec41bc51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016b08e4-7659-4907-9ade-5828ec41bc51-maas-api-openshift-ca\") pod \"016b08e4-7659-4907-9ade-5828ec41bc51\" (UID: \"016b08e4-7659-4907-9ade-5828ec41bc51\") "
Oct 30 23:54:05.287208 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.286777    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/016b08e4-7659-4907-9ade-5828ec41bc51-kube-api-access-sqhw8\" (UniqueName: \"kubernetes.io/projected/016b08e4-7659-4907-9ade-5828ec41bc51-kube-api-access-sqhw8\") pod \"016b08e4-7659-4907-9ade-5828ec41bc51\" (UID: \"016b08e4-7659-4907-9ade-5828ec41bc51\") "
Oct 30 23:54:05.287208 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.286963    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-maas-api-openshift-ca" pod "1a6bf5bf-83fc-4be2-b5d0-a42a6f821132" (UID: "1a6bf5bf-83fc-4be2-b5d0-a42a6f821132"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:54:05.287208 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.287129    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/016b08e4-7659-4907-9ade-5828ec41bc51-maas-api-openshift-ca" pod "016b08e4-7659-4907-9ade-5828ec41bc51" (UID: "016b08e4-7659-4907-9ade-5828ec41bc51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:54:05.287672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.287248    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e4edba-2552-47c2-bf82-69f0f6f2482f-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:05.287672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.287268    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/016b08e4-7659-4907-9ade-5828ec41bc51-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:05.287672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.287285    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j855d\" (UniqueName: \"kubernetes.io/projected/a9e4edba-2552-47c2-bf82-69f0f6f2482f-kube-api-access-j855d\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:05.287672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.287300    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:05.289004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.288968    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-maas-api-xks-ca" pod "1a6bf5bf-83fc-4be2-b5d0-a42a6f821132" (UID: "1a6bf5bf-83fc-4be2-b5d0-a42a6f821132"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:54:05.289099 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.289007    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-kube-api-access-pcl9s" pod "1a6bf5bf-83fc-4be2-b5d0-a42a6f821132" (UID: "1a6bf5bf-83fc-4be2-b5d0-a42a6f821132"). InnerVolumeSpecName "kube-api-access-pcl9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:54:05.289099 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.289055    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/016b08e4-7659-4907-9ade-5828ec41bc51-maas-api-xks-ca" pod "016b08e4-7659-4907-9ade-5828ec41bc51" (UID: "016b08e4-7659-4907-9ade-5828ec41bc51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:54:05.289099 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.289068    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/016b08e4-7659-4907-9ade-5828ec41bc51-kube-api-access-sqhw8" pod "016b08e4-7659-4907-9ade-5828ec41bc51" (UID: "016b08e4-7659-4907-9ade-5828ec41bc51"). InnerVolumeSpecName "kube-api-access-sqhw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:54:05.388196 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.388166    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqhw8\" (UniqueName: \"kubernetes.io/projected/016b08e4-7659-4907-9ade-5828ec41bc51-kube-api-access-sqhw8\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:05.388196 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.388195    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcl9s\" (UniqueName: \"kubernetes.io/projected/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-kube-api-access-pcl9s\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:05.388409 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.388209    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:05.388409 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:05.388221    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/016b08e4-7659-4907-9ade-5828ec41bc51-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:06.033218 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:06.033188    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt"
Oct 30 23:54:06.033382 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:06.033220    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt" event={"ID":"a9e4edba-2552-47c2-bf82-69f0f6f2482f","Type":"ContainerDied","Data":"2d11a95def1f74481957ab099c5f4bb264c9a045a20a010df1ea9b37ba04a1f5"}
Oct 30 23:54:06.033382 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:06.033272    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d11a95def1f74481957ab099c5f4bb264c9a045a20a010df1ea9b37ba04a1f5"
Oct 30 23:54:06.034685 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:06.034657    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn" event={"ID":"1a6bf5bf-83fc-4be2-b5d0-a42a6f821132","Type":"ContainerDied","Data":"d24bace7799eab49406d8523f5574d5f4643b12876397f6396f68b8864024549"}
Oct 30 23:54:06.034830 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:06.034700    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d24bace7799eab49406d8523f5574d5f4643b12876397f6396f68b8864024549"
Oct 30 23:54:06.034830 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:06.034664    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn"
Oct 30 23:54:06.036271 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:06.036254    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg"
Oct 30 23:54:06.036419 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:06.036271    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg" event={"ID":"016b08e4-7659-4907-9ade-5828ec41bc51","Type":"ContainerDied","Data":"98a8cfc86793aa3edc3b5de4ece650f2553218de91213ae553a35d83cce7c292"}
Oct 30 23:54:06.036419 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:06.036314    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98a8cfc86793aa3edc3b5de4ece650f2553218de91213ae553a35d83cce7c292"
Oct 30 23:54:07.768060 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:07.768012    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h" podUID="bdf11a9b-5acc-40dd-86e8-8fde0cd486d3" containerName="main" containerID="cri-o://82e132c2cfbc1284fc710e6ee11db1236cb40ed034244fd1d71d02dbb294c02b" gracePeriod=45
Oct 30 23:54:07.932272 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:07.932231    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h" podUID="bdf11a9b-5acc-40dd-86e8-8fde0cd486d3" containerName="main" probeResult="failure" output="Get \"https://10.133.0.227:8000/ready\": dial tcp 10.133.0.227:8000: connect: connection refused"
Oct 30 23:54:08.029822 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.029765    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:54:08.050162 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.050137    5078 generic.go:356] "Generic (PLEG): container finished" podID="bdf11a9b-5acc-40dd-86e8-8fde0cd486d3" containerID="82e132c2cfbc1284fc710e6ee11db1236cb40ed034244fd1d71d02dbb294c02b" exitCode=0
Oct 30 23:54:08.050300 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.050241    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h" event={"ID":"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3","Type":"ContainerDied","Data":"82e132c2cfbc1284fc710e6ee11db1236cb40ed034244fd1d71d02dbb294c02b"}
Oct 30 23:54:08.050300 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.050253    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"
Oct 30 23:54:08.050300 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.050274    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h" event={"ID":"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3","Type":"ContainerDied","Data":"05a5eb170e8dc5ee7fda9c501a4279b69a42c21995af35f22fbb29b0a9325c47"}
Oct 30 23:54:08.050300 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.050296    5078 scope.go:122] "RemoveContainer" containerID="82e132c2cfbc1284fc710e6ee11db1236cb40ed034244fd1d71d02dbb294c02b"
Oct 30 23:54:08.059052 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.059029    5078 scope.go:122] "RemoveContainer" containerID="82e132c2cfbc1284fc710e6ee11db1236cb40ed034244fd1d71d02dbb294c02b"
Oct 30 23:54:08.059328 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:54:08.059310    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82e132c2cfbc1284fc710e6ee11db1236cb40ed034244fd1d71d02dbb294c02b\": container with ID starting with 82e132c2cfbc1284fc710e6ee11db1236cb40ed034244fd1d71d02dbb294c02b not found: ID does not exist" containerID="82e132c2cfbc1284fc710e6ee11db1236cb40ed034244fd1d71d02dbb294c02b"
Oct 30 23:54:08.059385 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.059338    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82e132c2cfbc1284fc710e6ee11db1236cb40ed034244fd1d71d02dbb294c02b"} err="failed to get container status \"82e132c2cfbc1284fc710e6ee11db1236cb40ed034244fd1d71d02dbb294c02b\": rpc error: code = NotFound desc = could not find container \"82e132c2cfbc1284fc710e6ee11db1236cb40ed034244fd1d71d02dbb294c02b\": container with ID starting with 82e132c2cfbc1284fc710e6ee11db1236cb40ed034244fd1d71d02dbb294c02b not found: ID does not exist"
Oct 30 23:54:08.118285 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.118259    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-model-cache\") pod \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") "
Oct 30 23:54:08.118285 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.118288    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-dshm\") pod \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") "
Oct 30 23:54:08.118479 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.118356    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-home\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-home\") pod \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") "
Oct 30 23:54:08.118479 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.118397    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-kube-api-access-7xcfw\" (UniqueName: \"kubernetes.io/projected/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-kube-api-access-7xcfw\") pod \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") "
Oct 30 23:54:08.118479 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.118415    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-tls-certs\") pod \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") "
Oct 30 23:54:08.118479 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.118476    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-tmp-dir\") pod \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\" (UID: \"bdf11a9b-5acc-40dd-86e8-8fde0cd486d3\") "
Oct 30 23:54:08.118740 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.118554    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-model-cache" pod "bdf11a9b-5acc-40dd-86e8-8fde0cd486d3" (UID: "bdf11a9b-5acc-40dd-86e8-8fde0cd486d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:54:08.118740 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.118664    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-home" pod "bdf11a9b-5acc-40dd-86e8-8fde0cd486d3" (UID: "bdf11a9b-5acc-40dd-86e8-8fde0cd486d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:54:08.118868 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.118824    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-tmp-dir" pod "bdf11a9b-5acc-40dd-86e8-8fde0cd486d3" (UID: "bdf11a9b-5acc-40dd-86e8-8fde0cd486d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:54:08.118982 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.118964    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:08.119050 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.118988    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:08.119050 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.119004    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:08.120700 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.120680    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-dshm" pod "bdf11a9b-5acc-40dd-86e8-8fde0cd486d3" (UID: "bdf11a9b-5acc-40dd-86e8-8fde0cd486d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:54:08.120799 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.120757    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-tls-certs" pod "bdf11a9b-5acc-40dd-86e8-8fde0cd486d3" (UID: "bdf11a9b-5acc-40dd-86e8-8fde0cd486d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:54:08.120799 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.120782    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-kube-api-access-7xcfw" pod "bdf11a9b-5acc-40dd-86e8-8fde0cd486d3" (UID: "bdf11a9b-5acc-40dd-86e8-8fde0cd486d3"). InnerVolumeSpecName "kube-api-access-7xcfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:54:08.220007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.219974    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:08.220007 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.220005    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xcfw\" (UniqueName: \"kubernetes.io/projected/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-kube-api-access-7xcfw\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:08.220198 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.220020    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:08.463780 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.463740    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"]
Oct 30 23:54:08.466802 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:08.466776    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5e3f9d/test-model-5e3f9d-kserve-6fd5c9f455-cmh2h"]
Oct 30 23:54:10.298089 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:10.298054    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdf11a9b-5acc-40dd-86e8-8fde0cd486d3" path="/var/lib/kubelet/pods/bdf11a9b-5acc-40dd-86e8-8fde0cd486d3/volumes"
Oct 30 23:54:17.677084 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:54:17.677046    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:54:35.300433 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.300400    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22"]
Oct 30 23:54:35.300862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.300854    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a44f9d90-59b0-4e9d-b95e-e987edcb1266" containerName="payload-processing"
Oct 30 23:54:35.300906 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.300867    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44f9d90-59b0-4e9d-b95e-e987edcb1266" containerName="payload-processing"
Oct 30 23:54:35.300906 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.300881    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8" containerName="payload-pre-processing"
Oct 30 23:54:35.300906 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.300885    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8" containerName="payload-pre-processing"
Oct 30 23:54:35.300906 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.300895    5078 cpu_manager.go:414] "containerMap: removing container" podUID="016b08e4-7659-4907-9ade-5828ec41bc51" containerName="revoke-keys"
Oct 30 23:54:35.300906 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.300900    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="016b08e4-7659-4907-9ade-5828ec41bc51" containerName="revoke-keys"
Oct 30 23:54:35.301063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.300911    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a9e4edba-2552-47c2-bf82-69f0f6f2482f" containerName="revoke-keys"
Oct 30 23:54:35.301063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.300916    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e4edba-2552-47c2-bf82-69f0f6f2482f" containerName="revoke-keys"
Oct 30 23:54:35.301063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.300935    5078 cpu_manager.go:414] "containerMap: removing container" podUID="bdf11a9b-5acc-40dd-86e8-8fde0cd486d3" containerName="main"
Oct 30 23:54:35.301063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.300940    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf11a9b-5acc-40dd-86e8-8fde0cd486d3" containerName="main"
Oct 30 23:54:35.301063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.300958    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1a6bf5bf-83fc-4be2-b5d0-a42a6f821132" containerName="revoke-keys"
Oct 30 23:54:35.301063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.300963    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a6bf5bf-83fc-4be2-b5d0-a42a6f821132" containerName="revoke-keys"
Oct 30 23:54:35.301063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.301045    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44f9d90-59b0-4e9d-b95e-e987edcb1266" containerName="payload-processing"
Oct 30 23:54:35.301063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.301055    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d56ab6e-b5b6-4c2e-bdf1-676d50b57fd8" containerName="payload-pre-processing"
Oct 30 23:54:35.301063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.301060    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="016b08e4-7659-4907-9ade-5828ec41bc51" containerName="revoke-keys"
Oct 30 23:54:35.301332 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.301067    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e4edba-2552-47c2-bf82-69f0f6f2482f" containerName="revoke-keys"
Oct 30 23:54:35.301332 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.301074    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf11a9b-5acc-40dd-86e8-8fde0cd486d3" containerName="main"
Oct 30 23:54:35.301332 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.301087    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a6bf5bf-83fc-4be2-b5d0-a42a6f821132" containerName="revoke-keys"
Oct 30 23:54:35.304252 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.304231    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22"
Oct 30 23:54:35.358931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.358836    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:54:35.408148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.408121    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:54:35.408148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.408146    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:54:35.408148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.408156    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:54:35.410690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.410672    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22"]
Oct 30 23:54:35.432899 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.432880    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22\" (UID: \"c0c9cf39-6d09-4a15-b5fb-18afab3743c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22"
Oct 30 23:54:35.432980 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.432917    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22\" (UID: \"c0c9cf39-6d09-4a15-b5fb-18afab3743c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22"
Oct 30 23:54:35.433103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.433085    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlx4v\" (UniqueName: \"kubernetes.io/projected/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-kube-api-access-vlx4v\") pod \"maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22\" (UID: \"c0c9cf39-6d09-4a15-b5fb-18afab3743c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22"
Oct 30 23:54:35.534228 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.534177    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlx4v\" (UniqueName: \"kubernetes.io/projected/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-kube-api-access-vlx4v\") pod \"maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22\" (UID: \"c0c9cf39-6d09-4a15-b5fb-18afab3743c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22"
Oct 30 23:54:35.534423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.534313    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22\" (UID: \"c0c9cf39-6d09-4a15-b5fb-18afab3743c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22"
Oct 30 23:54:35.534423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.534356    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22\" (UID: \"c0c9cf39-6d09-4a15-b5fb-18afab3743c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22"
Oct 30 23:54:35.534994 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.534972    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22\" (UID: \"c0c9cf39-6d09-4a15-b5fb-18afab3743c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22"
Oct 30 23:54:35.537060 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.537026    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22\" (UID: \"c0c9cf39-6d09-4a15-b5fb-18afab3743c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22"
Oct 30 23:54:35.592175 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.592106    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlx4v\" (UniqueName: \"kubernetes.io/projected/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-kube-api-access-vlx4v\") pod \"maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22\" (UID: \"c0c9cf39-6d09-4a15-b5fb-18afab3743c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22"
Oct 30 23:54:35.616815 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.616785    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22"
Oct 30 23:54:35.987478 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:35.987452    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22"]
Oct 30 23:54:35.988779 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:54:35.988754    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c9cf39_6d09_4a15_b5fb_18afab3743c9.slice/crio-d02f456b908de956bd02012ceba44a4653797c41dbd855b85bbff073f1771b7a WatchSource:0}: Error finding container d02f456b908de956bd02012ceba44a4653797c41dbd855b85bbff073f1771b7a: Status 404 returned error can't find the container with id d02f456b908de956bd02012ceba44a4653797c41dbd855b85bbff073f1771b7a
Oct 30 23:54:36.216957 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:36.216922    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22" event={"ID":"c0c9cf39-6d09-4a15-b5fb-18afab3743c9","Type":"ContainerStarted","Data":"d02f456b908de956bd02012ceba44a4653797c41dbd855b85bbff073f1771b7a"}
Oct 30 23:54:37.225807 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:37.225722    5078 generic.go:356] "Generic (PLEG): container finished" podID="c0c9cf39-6d09-4a15-b5fb-18afab3743c9" containerID="3e2f8b507638e2e6108c4cb529dfb38ba0f06cfe617e542f54427f693065d812" exitCode=0
Oct 30 23:54:37.226209 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:37.225814    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22" event={"ID":"c0c9cf39-6d09-4a15-b5fb-18afab3743c9","Type":"ContainerDied","Data":"3e2f8b507638e2e6108c4cb529dfb38ba0f06cfe617e542f54427f693065d812"}
Oct 30 23:54:37.680339 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:37.680303    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"]
Oct 30 23:54:38.371609 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:38.371571    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22"
Oct 30 23:54:38.463241 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:38.463215    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-kube-api-access-vlx4v\" (UniqueName: \"kubernetes.io/projected/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-kube-api-access-vlx4v\") pod \"c0c9cf39-6d09-4a15-b5fb-18afab3743c9\" (UID: \"c0c9cf39-6d09-4a15-b5fb-18afab3743c9\") "
Oct 30 23:54:38.463389 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:38.463308    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-maas-api-xks-ca\") pod \"c0c9cf39-6d09-4a15-b5fb-18afab3743c9\" (UID: \"c0c9cf39-6d09-4a15-b5fb-18afab3743c9\") "
Oct 30 23:54:38.463389 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:38.463374    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-maas-api-openshift-ca\") pod \"c0c9cf39-6d09-4a15-b5fb-18afab3743c9\" (UID: \"c0c9cf39-6d09-4a15-b5fb-18afab3743c9\") "
Oct 30 23:54:38.463718 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:38.463698    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-maas-api-openshift-ca" pod "c0c9cf39-6d09-4a15-b5fb-18afab3743c9" (UID: "c0c9cf39-6d09-4a15-b5fb-18afab3743c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:54:38.463881 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:38.463861    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:38.465621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:38.465581    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-kube-api-access-vlx4v" pod "c0c9cf39-6d09-4a15-b5fb-18afab3743c9" (UID: "c0c9cf39-6d09-4a15-b5fb-18afab3743c9"). InnerVolumeSpecName "kube-api-access-vlx4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:54:38.465710 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:38.465686    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-maas-api-xks-ca" pod "c0c9cf39-6d09-4a15-b5fb-18afab3743c9" (UID: "c0c9cf39-6d09-4a15-b5fb-18afab3743c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:54:38.564914 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:38.564880    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlx4v\" (UniqueName: \"kubernetes.io/projected/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-kube-api-access-vlx4v\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:38.564914 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:38.564909    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c9cf39-6d09-4a15-b5fb-18afab3743c9-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:39.239213 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:39.239182    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22"
Oct 30 23:54:39.239527 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:39.239186    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22" event={"ID":"c0c9cf39-6d09-4a15-b5fb-18afab3743c9","Type":"ContainerDied","Data":"d02f456b908de956bd02012ceba44a4653797c41dbd855b85bbff073f1771b7a"}
Oct 30 23:54:39.239527 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:39.239303    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d02f456b908de956bd02012ceba44a4653797c41dbd855b85bbff073f1771b7a"
Oct 30 23:54:43.193890 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.193857    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq"]
Oct 30 23:54:43.194373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.194357    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c0c9cf39-6d09-4a15-b5fb-18afab3743c9" containerName="revoke-keys"
Oct 30 23:54:43.194423 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.194375    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c9cf39-6d09-4a15-b5fb-18afab3743c9" containerName="revoke-keys"
Oct 30 23:54:43.194508 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.194498    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c9cf39-6d09-4a15-b5fb-18afab3743c9" containerName="revoke-keys"
Oct 30 23:54:43.197937 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.197920    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq"
Oct 30 23:54:43.252165 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.251820    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:54:43.252165 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.252046    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:54:43.252165 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.252126    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:54:43.252165 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.252133    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:54:43.255282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.255258    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq"]
Oct 30 23:54:43.315532 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.315494    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt9jw\" (UniqueName: \"kubernetes.io/projected/f751e5aa-c6df-493f-9ecd-0756d593c0ef-kube-api-access-lt9jw\") pod \"maas-api-revoke-sub-5ed97f4b-7zsxq\" (UID: \"f751e5aa-c6df-493f-9ecd-0756d593c0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq"
Oct 30 23:54:43.315727 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.315537    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f751e5aa-c6df-493f-9ecd-0756d593c0ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ed97f4b-7zsxq\" (UID: \"f751e5aa-c6df-493f-9ecd-0756d593c0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq"
Oct 30 23:54:43.315779 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.315737    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f751e5aa-c6df-493f-9ecd-0756d593c0ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ed97f4b-7zsxq\" (UID: \"f751e5aa-c6df-493f-9ecd-0756d593c0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq"
Oct 30 23:54:43.416682 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.416653    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9jw\" (UniqueName: \"kubernetes.io/projected/f751e5aa-c6df-493f-9ecd-0756d593c0ef-kube-api-access-lt9jw\") pod \"maas-api-revoke-sub-5ed97f4b-7zsxq\" (UID: \"f751e5aa-c6df-493f-9ecd-0756d593c0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq"
Oct 30 23:54:43.416682 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.416685    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f751e5aa-c6df-493f-9ecd-0756d593c0ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ed97f4b-7zsxq\" (UID: \"f751e5aa-c6df-493f-9ecd-0756d593c0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq"
Oct 30 23:54:43.416948 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.416788    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f751e5aa-c6df-493f-9ecd-0756d593c0ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ed97f4b-7zsxq\" (UID: \"f751e5aa-c6df-493f-9ecd-0756d593c0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq"
Oct 30 23:54:43.417376 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.417352    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f751e5aa-c6df-493f-9ecd-0756d593c0ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ed97f4b-7zsxq\" (UID: \"f751e5aa-c6df-493f-9ecd-0756d593c0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq"
Oct 30 23:54:43.419272 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.419249    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f751e5aa-c6df-493f-9ecd-0756d593c0ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ed97f4b-7zsxq\" (UID: \"f751e5aa-c6df-493f-9ecd-0756d593c0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq"
Oct 30 23:54:43.474563 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.474510    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt9jw\" (UniqueName: \"kubernetes.io/projected/f751e5aa-c6df-493f-9ecd-0756d593c0ef-kube-api-access-lt9jw\") pod \"maas-api-revoke-sub-5ed97f4b-7zsxq\" (UID: \"f751e5aa-c6df-493f-9ecd-0756d593c0ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq"
Oct 30 23:54:43.510491 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.510465    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq"
Oct 30 23:54:43.881905 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:43.881879    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq"]
Oct 30 23:54:43.883835 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:54:43.883813    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf751e5aa_c6df_493f_9ecd_0756d593c0ef.slice/crio-c77cf2b42e66ce7cd07cbd86e26e57d30a903b03b926916922aaf700ab28f44c WatchSource:0}: Error finding container c77cf2b42e66ce7cd07cbd86e26e57d30a903b03b926916922aaf700ab28f44c: Status 404 returned error can't find the container with id c77cf2b42e66ce7cd07cbd86e26e57d30a903b03b926916922aaf700ab28f44c
Oct 30 23:54:44.271763 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:44.271735    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq" event={"ID":"f751e5aa-c6df-493f-9ecd-0756d593c0ef","Type":"ContainerStarted","Data":"ffd748afaddada7c17c514a1825e2dac966b9c1ec254e9c1857162658854f4d0"}
Oct 30 23:54:44.272250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:44.271781    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq" event={"ID":"f751e5aa-c6df-493f-9ecd-0756d593c0ef","Type":"ContainerStarted","Data":"c77cf2b42e66ce7cd07cbd86e26e57d30a903b03b926916922aaf700ab28f44c"}
Oct 30 23:54:44.386384 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:44.386331    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq" podStartSLOduration=1.38631759 podStartE2EDuration="1.38631759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:44.384902615 +0000 UTC m=+2870.272808980" watchObservedRunningTime="2026-10-30 23:54:44.38631759 +0000 UTC m=+2870.274223947"
Oct 30 23:54:45.279215 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:45.279136    5078 generic.go:356] "Generic (PLEG): container finished" podID="f751e5aa-c6df-493f-9ecd-0756d593c0ef" containerID="ffd748afaddada7c17c514a1825e2dac966b9c1ec254e9c1857162658854f4d0" exitCode=0
Oct 30 23:54:45.279660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:45.279217    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq" event={"ID":"f751e5aa-c6df-493f-9ecd-0756d593c0ef","Type":"ContainerDied","Data":"ffd748afaddada7c17c514a1825e2dac966b9c1ec254e9c1857162658854f4d0"}
Oct 30 23:54:46.437698 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:46.437677    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq"
Oct 30 23:54:46.554297 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:46.554266    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f751e5aa-c6df-493f-9ecd-0756d593c0ef-kube-api-access-lt9jw\" (UniqueName: \"kubernetes.io/projected/f751e5aa-c6df-493f-9ecd-0756d593c0ef-kube-api-access-lt9jw\") pod \"f751e5aa-c6df-493f-9ecd-0756d593c0ef\" (UID: \"f751e5aa-c6df-493f-9ecd-0756d593c0ef\") "
Oct 30 23:54:46.554505 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:46.554398    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f751e5aa-c6df-493f-9ecd-0756d593c0ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f751e5aa-c6df-493f-9ecd-0756d593c0ef-maas-api-xks-ca\") pod \"f751e5aa-c6df-493f-9ecd-0756d593c0ef\" (UID: \"f751e5aa-c6df-493f-9ecd-0756d593c0ef\") "
Oct 30 23:54:46.554588 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:46.554518    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f751e5aa-c6df-493f-9ecd-0756d593c0ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f751e5aa-c6df-493f-9ecd-0756d593c0ef-maas-api-openshift-ca\") pod \"f751e5aa-c6df-493f-9ecd-0756d593c0ef\" (UID: \"f751e5aa-c6df-493f-9ecd-0756d593c0ef\") "
Oct 30 23:54:46.555461 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:46.555432    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f751e5aa-c6df-493f-9ecd-0756d593c0ef-maas-api-openshift-ca" pod "f751e5aa-c6df-493f-9ecd-0756d593c0ef" (UID: "f751e5aa-c6df-493f-9ecd-0756d593c0ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:54:46.556884 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:46.556860    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f751e5aa-c6df-493f-9ecd-0756d593c0ef-kube-api-access-lt9jw" pod "f751e5aa-c6df-493f-9ecd-0756d593c0ef" (UID: "f751e5aa-c6df-493f-9ecd-0756d593c0ef"). InnerVolumeSpecName "kube-api-access-lt9jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:54:46.557195 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:46.557169    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f751e5aa-c6df-493f-9ecd-0756d593c0ef-maas-api-xks-ca" pod "f751e5aa-c6df-493f-9ecd-0756d593c0ef" (UID: "f751e5aa-c6df-493f-9ecd-0756d593c0ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:54:46.656859 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:46.656824    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f751e5aa-c6df-493f-9ecd-0756d593c0ef-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:46.656859 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:46.656861    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f751e5aa-c6df-493f-9ecd-0756d593c0ef-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:46.657078 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:46.656876    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt9jw\" (UniqueName: \"kubernetes.io/projected/f751e5aa-c6df-493f-9ecd-0756d593c0ef-kube-api-access-lt9jw\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:47.291045 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:47.291014    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq"
Oct 30 23:54:47.291045 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:47.291029    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq" event={"ID":"f751e5aa-c6df-493f-9ecd-0756d593c0ef","Type":"ContainerDied","Data":"c77cf2b42e66ce7cd07cbd86e26e57d30a903b03b926916922aaf700ab28f44c"}
Oct 30 23:54:47.291304 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:47.291070    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c77cf2b42e66ce7cd07cbd86e26e57d30a903b03b926916922aaf700ab28f44c"
Oct 30 23:54:47.590507 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:54:47.590425    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:54:52.701065 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:52.701017    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7" podUID="96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc" containerName="main" containerID="cri-o://f60fe5a37e7e36572a2e74a483548b1aeb7db696fb3f97805b96b77ad2ffb1ae" gracePeriod=45
Oct 30 23:54:52.970790 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:52.970762    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:54:53.027926 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.027894    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-dshm\") pod \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") "
Oct 30 23:54:53.028121 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.027959    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-tls-certs\") pod \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") "
Oct 30 23:54:53.028121 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.027988    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-tmp-dir\") pod \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") "
Oct 30 23:54:53.028121 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.028015    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-home\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-home\") pod \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") "
Oct 30 23:54:53.028277 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.028195    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-kube-api-access-4qds4\" (UniqueName: \"kubernetes.io/projected/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-kube-api-access-4qds4\") pod \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") "
Oct 30 23:54:53.028332 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.028286    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-home" pod "96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc" (UID: "96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:54:53.028332 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.028289    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-model-cache\") pod \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\" (UID: \"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc\") "
Oct 30 23:54:53.028431 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.028300    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-tmp-dir" pod "96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc" (UID: "96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:54:53.028973 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.028946    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:53.028973 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.028974    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:53.029429 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.029402    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-model-cache" pod "96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc" (UID: "96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:54:53.030793 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.030689    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-dshm" pod "96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc" (UID: "96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:54:53.030793 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.030726    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-kube-api-access-4qds4" pod "96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc" (UID: "96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc"). InnerVolumeSpecName "kube-api-access-4qds4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:54:53.030793 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.030783    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-tls-certs" pod "96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc" (UID: "96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:54:53.129982 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.129952    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:53.129982 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.129976    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:53.129982 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.129987    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qds4\" (UniqueName: \"kubernetes.io/projected/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-kube-api-access-4qds4\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:53.130248 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.129996    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:54:53.326419 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.326380    5078 generic.go:356] "Generic (PLEG): container finished" podID="96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc" containerID="f60fe5a37e7e36572a2e74a483548b1aeb7db696fb3f97805b96b77ad2ffb1ae" exitCode=0
Oct 30 23:54:53.326642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.326445    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"
Oct 30 23:54:53.326642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.326477    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7" event={"ID":"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc","Type":"ContainerDied","Data":"f60fe5a37e7e36572a2e74a483548b1aeb7db696fb3f97805b96b77ad2ffb1ae"}
Oct 30 23:54:53.326642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.326537    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7" event={"ID":"96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc","Type":"ContainerDied","Data":"002fb914e98585ee9f8306283b4b4023139b418105d2670ffc1806196604eb42"}
Oct 30 23:54:53.326642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.326563    5078 scope.go:122] "RemoveContainer" containerID="f60fe5a37e7e36572a2e74a483548b1aeb7db696fb3f97805b96b77ad2ffb1ae"
Oct 30 23:54:53.337160 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.337142    5078 scope.go:122] "RemoveContainer" containerID="f60fe5a37e7e36572a2e74a483548b1aeb7db696fb3f97805b96b77ad2ffb1ae"
Oct 30 23:54:53.337522 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:54:53.337495    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f60fe5a37e7e36572a2e74a483548b1aeb7db696fb3f97805b96b77ad2ffb1ae\": container with ID starting with f60fe5a37e7e36572a2e74a483548b1aeb7db696fb3f97805b96b77ad2ffb1ae not found: ID does not exist" containerID="f60fe5a37e7e36572a2e74a483548b1aeb7db696fb3f97805b96b77ad2ffb1ae"
Oct 30 23:54:53.337664 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.337528    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f60fe5a37e7e36572a2e74a483548b1aeb7db696fb3f97805b96b77ad2ffb1ae"} err="failed to get container status \"f60fe5a37e7e36572a2e74a483548b1aeb7db696fb3f97805b96b77ad2ffb1ae\": rpc error: code = NotFound desc = could not find container \"f60fe5a37e7e36572a2e74a483548b1aeb7db696fb3f97805b96b77ad2ffb1ae\": container with ID starting with f60fe5a37e7e36572a2e74a483548b1aeb7db696fb3f97805b96b77ad2ffb1ae not found: ID does not exist"
Oct 30 23:54:53.495855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.495820    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"]
Oct 30 23:54:53.541369 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:53.541342    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-15b137/test-model-15b137-kserve-6fb8db7765-t8rs7"]
Oct 30 23:54:54.299651 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:54:54.299585    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc" path="/var/lib/kubelet/pods/96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc/volumes"
Oct 30 23:55:03.735395 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:03.735320    5078 scope.go:122] "RemoveContainer" containerID="27612c6f5f5b7dc217b7ed0c4c0397bd625a1504fe3f319286d121b5f882f4a6"
Oct 30 23:55:03.744079 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:03.744059    5078 scope.go:122] "RemoveContainer" containerID="415f60846e23484e5c9735aca92f31df1164e95c4df0c232f168f510bd78f568"
Oct 30 23:55:03.752056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:03.752039    5078 scope.go:122] "RemoveContainer" containerID="42bc6256b70b7bb98570e0a973f171533441320fc5d9752897d5c2a4e50915b8"
Oct 30 23:55:08.152905 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:08.152868    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2"]
Oct 30 23:55:08.202587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:08.202552    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-2cfb80-r6kq2"]
Oct 30 23:55:08.298261 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:08.298229    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9badfba-c1db-425e-aec7-7a6aa7838d27" path="/var/lib/kubelet/pods/c9badfba-c1db-425e-aec7-7a6aa7838d27/volumes"
Oct 30 23:55:09.399930 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:09.399892    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22"]
Oct 30 23:55:09.449606 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:09.449573    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5e3f9d-7cs22"]
Oct 30 23:55:09.796105 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:09.796070    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9"]
Oct 30 23:55:09.796485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:09.796472    5078 cpu_manager.go:414] "containerMap: removing container" podUID="96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc" containerName="main"
Oct 30 23:55:09.796546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:09.796486    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc" containerName="main"
Oct 30 23:55:09.796546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:09.796497    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f751e5aa-c6df-493f-9ecd-0756d593c0ef" containerName="revoke-keys"
Oct 30 23:55:09.796546 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:09.796502    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f751e5aa-c6df-493f-9ecd-0756d593c0ef" containerName="revoke-keys"
Oct 30 23:55:09.796704 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:09.796587    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a3c9e3-edeb-4f10-8b80-7d42fd84b3dc" containerName="main"
Oct 30 23:55:09.796704 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:09.796612    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f751e5aa-c6df-493f-9ecd-0756d593c0ef" containerName="revoke-keys"
Oct 30 23:55:09.801378 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:09.801362    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9"
Oct 30 23:55:09.860167 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:09.860098    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:55:09.860472 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:09.860454    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:55:09.862820 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:09.862796    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9"]
Oct 30 23:55:09.909115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:09.909089    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:55:09.909115 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:09.909089    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:55:09.928114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:09.928084    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9\" (UID: \"1e09f1b6-3e74-408b-8118-e68da0ca5bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9"
Oct 30 23:55:09.928258 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:09.928191    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lggzk\" (UniqueName: \"kubernetes.io/projected/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-kube-api-access-lggzk\") pod \"maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9\" (UID: \"1e09f1b6-3e74-408b-8118-e68da0ca5bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9"
Oct 30 23:55:09.928360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:09.928342    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9\" (UID: \"1e09f1b6-3e74-408b-8118-e68da0ca5bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9"
Oct 30 23:55:10.030065 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:10.030035    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9\" (UID: \"1e09f1b6-3e74-408b-8118-e68da0ca5bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9"
Oct 30 23:55:10.030205 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:10.030130    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9\" (UID: \"1e09f1b6-3e74-408b-8118-e68da0ca5bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9"
Oct 30 23:55:10.030205 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:10.030162    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lggzk\" (UniqueName: \"kubernetes.io/projected/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-kube-api-access-lggzk\") pod \"maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9\" (UID: \"1e09f1b6-3e74-408b-8118-e68da0ca5bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9"
Oct 30 23:55:10.030889 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:10.030776    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9\" (UID: \"1e09f1b6-3e74-408b-8118-e68da0ca5bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9"
Oct 30 23:55:10.032633 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:10.032609    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9\" (UID: \"1e09f1b6-3e74-408b-8118-e68da0ca5bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9"
Oct 30 23:55:10.088104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:10.088049    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lggzk\" (UniqueName: \"kubernetes.io/projected/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-kube-api-access-lggzk\") pod \"maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9\" (UID: \"1e09f1b6-3e74-408b-8118-e68da0ca5bbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9"
Oct 30 23:55:10.113661 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:10.113640    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9"
Oct 30 23:55:10.298050 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:10.298019    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c9cf39-6d09-4a15-b5fb-18afab3743c9" path="/var/lib/kubelet/pods/c0c9cf39-6d09-4a15-b5fb-18afab3743c9/volumes"
Oct 30 23:55:10.493116 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:10.493094    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9"]
Oct 30 23:55:10.495096 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:55:10.495067    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e09f1b6_3e74_408b_8118_e68da0ca5bbb.slice/crio-2010be5c410865952284cbf1115ebbc7b20a32decf0cb55b87476d310c5b2aab WatchSource:0}: Error finding container 2010be5c410865952284cbf1115ebbc7b20a32decf0cb55b87476d310c5b2aab: Status 404 returned error can't find the container with id 2010be5c410865952284cbf1115ebbc7b20a32decf0cb55b87476d310c5b2aab
Oct 30 23:55:11.451738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:11.451701    5078 generic.go:356] "Generic (PLEG): container finished" podID="1e09f1b6-3e74-408b-8118-e68da0ca5bbb" containerID="60dfcbc499a57a0238d7d00eaae75af77cc046c0b16f1ca32ed5b119fdf6feb4" exitCode=0
Oct 30 23:55:11.451912 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:11.451784    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9" event={"ID":"1e09f1b6-3e74-408b-8118-e68da0ca5bbb","Type":"ContainerDied","Data":"60dfcbc499a57a0238d7d00eaae75af77cc046c0b16f1ca32ed5b119fdf6feb4"}
Oct 30 23:55:11.451912 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:11.451836    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9" event={"ID":"1e09f1b6-3e74-408b-8118-e68da0ca5bbb","Type":"ContainerStarted","Data":"2010be5c410865952284cbf1115ebbc7b20a32decf0cb55b87476d310c5b2aab"}
Oct 30 23:55:12.594900 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:12.594880    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9"
Oct 30 23:55:12.759523 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:12.759452    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-maas-api-xks-ca\") pod \"1e09f1b6-3e74-408b-8118-e68da0ca5bbb\" (UID: \"1e09f1b6-3e74-408b-8118-e68da0ca5bbb\") "
Oct 30 23:55:12.759709 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:12.759567    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-maas-api-openshift-ca\") pod \"1e09f1b6-3e74-408b-8118-e68da0ca5bbb\" (UID: \"1e09f1b6-3e74-408b-8118-e68da0ca5bbb\") "
Oct 30 23:55:12.759709 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:12.759652    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-kube-api-access-lggzk\" (UniqueName: \"kubernetes.io/projected/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-kube-api-access-lggzk\") pod \"1e09f1b6-3e74-408b-8118-e68da0ca5bbb\" (UID: \"1e09f1b6-3e74-408b-8118-e68da0ca5bbb\") "
Oct 30 23:55:12.759925 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:12.759902    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-maas-api-openshift-ca" pod "1e09f1b6-3e74-408b-8118-e68da0ca5bbb" (UID: "1e09f1b6-3e74-408b-8118-e68da0ca5bbb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:55:12.760116 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:12.760100    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:55:12.761793 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:12.761765    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-kube-api-access-lggzk" pod "1e09f1b6-3e74-408b-8118-e68da0ca5bbb" (UID: "1e09f1b6-3e74-408b-8118-e68da0ca5bbb"). InnerVolumeSpecName "kube-api-access-lggzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:55:12.761885 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:12.761772    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-maas-api-xks-ca" pod "1e09f1b6-3e74-408b-8118-e68da0ca5bbb" (UID: "1e09f1b6-3e74-408b-8118-e68da0ca5bbb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:55:12.861532 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:12.861503    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lggzk\" (UniqueName: \"kubernetes.io/projected/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-kube-api-access-lggzk\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:55:12.861532 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:12.861531    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e09f1b6-3e74-408b-8118-e68da0ca5bbb-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:55:13.463566 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:13.463539    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9"
Oct 30 23:55:13.463792 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:13.463537    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9" event={"ID":"1e09f1b6-3e74-408b-8118-e68da0ca5bbb","Type":"ContainerDied","Data":"2010be5c410865952284cbf1115ebbc7b20a32decf0cb55b87476d310c5b2aab"}
Oct 30 23:55:13.463792 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:13.463647    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2010be5c410865952284cbf1115ebbc7b20a32decf0cb55b87476d310c5b2aab"
Oct 30 23:55:15.451160 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:15.451126    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"]
Oct 30 23:55:15.451556 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:15.451342    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z" podUID="b8c6ff7a-3974-4ca3-9580-4b83b87275e2" containerName="istio-proxy" containerID="cri-o://7766308982c9ee8dfcd04c7a61f171bc05b87efff47aea2db9702f614981a801" gracePeriod=30
Oct 30 23:55:17.125937 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:17.125890    5078 patch_prober.go:29] interesting pod/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:55:17.126344 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:17.125987    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z" podUID="b8c6ff7a-3974-4ca3-9580-4b83b87275e2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:55:17.591270 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:55:17.591238    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:55:18.016335 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.016254    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn"]
Oct 30 23:55:18.016757 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.016741    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1e09f1b6-3e74-408b-8118-e68da0ca5bbb" containerName="revoke-keys"
Oct 30 23:55:18.016854 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.016757    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e09f1b6-3e74-408b-8118-e68da0ca5bbb" containerName="revoke-keys"
Oct 30 23:55:18.016854 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.016849    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e09f1b6-3e74-408b-8118-e68da0ca5bbb" containerName="revoke-keys"
Oct 30 23:55:18.020168 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.020145    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn"
Oct 30 23:55:18.072832 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.072808    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:55:18.121649 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.121590    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/624ace8a-cc95-410a-b14e-905bbd8e76c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn\" (UID: \"624ace8a-cc95-410a-b14e-905bbd8e76c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn"
Oct 30 23:55:18.121834 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.121742    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/624ace8a-cc95-410a-b14e-905bbd8e76c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn\" (UID: \"624ace8a-cc95-410a-b14e-905bbd8e76c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn"
Oct 30 23:55:18.121936 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.121917    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29x4t\" (UniqueName: \"kubernetes.io/projected/624ace8a-cc95-410a-b14e-905bbd8e76c9-kube-api-access-29x4t\") pod \"maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn\" (UID: \"624ace8a-cc95-410a-b14e-905bbd8e76c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn"
Oct 30 23:55:18.122379 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.122360    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:55:18.122476 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.122390    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:55:18.122476 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.122412    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:55:18.125015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.124984    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn"]
Oct 30 23:55:18.223205 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.223175    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29x4t\" (UniqueName: \"kubernetes.io/projected/624ace8a-cc95-410a-b14e-905bbd8e76c9-kube-api-access-29x4t\") pod \"maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn\" (UID: \"624ace8a-cc95-410a-b14e-905bbd8e76c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn"
Oct 30 23:55:18.223711 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.223232    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/624ace8a-cc95-410a-b14e-905bbd8e76c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn\" (UID: \"624ace8a-cc95-410a-b14e-905bbd8e76c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn"
Oct 30 23:55:18.223711 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.223278    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/624ace8a-cc95-410a-b14e-905bbd8e76c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn\" (UID: \"624ace8a-cc95-410a-b14e-905bbd8e76c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn"
Oct 30 23:55:18.224083 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.224061    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/624ace8a-cc95-410a-b14e-905bbd8e76c9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn\" (UID: \"624ace8a-cc95-410a-b14e-905bbd8e76c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn"
Oct 30 23:55:18.225869 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.225843    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/624ace8a-cc95-410a-b14e-905bbd8e76c9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn\" (UID: \"624ace8a-cc95-410a-b14e-905bbd8e76c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn"
Oct 30 23:55:18.280465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.280410    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29x4t\" (UniqueName: \"kubernetes.io/projected/624ace8a-cc95-410a-b14e-905bbd8e76c9-kube-api-access-29x4t\") pod \"maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn\" (UID: \"624ace8a-cc95-410a-b14e-905bbd8e76c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn"
Oct 30 23:55:18.332534 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.332513    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn"
Oct 30 23:55:18.717437 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:18.717408    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn"]
Oct 30 23:55:18.718406 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:55:18.718379    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod624ace8a_cc95_410a_b14e_905bbd8e76c9.slice/crio-4678ea6127330b44ea193de8cfd1928da4046db3580ea26a300d0cc03ea550c5 WatchSource:0}: Error finding container 4678ea6127330b44ea193de8cfd1928da4046db3580ea26a300d0cc03ea550c5: Status 404 returned error can't find the container with id 4678ea6127330b44ea193de8cfd1928da4046db3580ea26a300d0cc03ea550c5
Oct 30 23:55:19.525044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:19.525004    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn" event={"ID":"624ace8a-cc95-410a-b14e-905bbd8e76c9","Type":"ContainerStarted","Data":"4b6d442a0cb5d72f65d7be987b400981d62edc1e82a7f940cd946016f34ebd4b"}
Oct 30 23:55:19.525448 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:19.525064    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn" event={"ID":"624ace8a-cc95-410a-b14e-905bbd8e76c9","Type":"ContainerStarted","Data":"4678ea6127330b44ea193de8cfd1928da4046db3580ea26a300d0cc03ea550c5"}
Oct 30 23:55:19.644371 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:19.644319    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn" podStartSLOduration=2.64430567 podStartE2EDuration="2.64430567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:55:19.641284424 +0000 UTC m=+2905.529190777" watchObservedRunningTime="2026-10-30 23:55:19.64430567 +0000 UTC m=+2905.532212022"
Oct 30 23:55:20.532652 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.532614    5078 generic.go:356] "Generic (PLEG): container finished" podID="624ace8a-cc95-410a-b14e-905bbd8e76c9" containerID="4b6d442a0cb5d72f65d7be987b400981d62edc1e82a7f940cd946016f34ebd4b" exitCode=0
Oct 30 23:55:20.533135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.532689    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn" event={"ID":"624ace8a-cc95-410a-b14e-905bbd8e76c9","Type":"ContainerDied","Data":"4b6d442a0cb5d72f65d7be987b400981d62edc1e82a7f940cd946016f34ebd4b"}
Oct 30 23:55:20.725144 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.725084    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:55:20.766935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.766908    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-credential-socket\") pod \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") "
Oct 30 23:55:20.767068 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.766962    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-workload-socket\") pod \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") "
Oct 30 23:55:20.767068 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.766991    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-workload-certs\") pod \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") "
Oct 30 23:55:20.767068 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.767050    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-kube-api-access-dvdwp\" (UniqueName: \"kubernetes.io/projected/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-kube-api-access-dvdwp\") pod \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") "
Oct 30 23:55:20.767251 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.767076    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-podinfo\") pod \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") "
Oct 30 23:55:20.767251 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.767114    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-token\" (UniqueName: \"kubernetes.io/projected/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-token\") pod \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") "
Oct 30 23:55:20.767251 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.767200    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-credential-socket" pod "b8c6ff7a-3974-4ca3-9580-4b83b87275e2" (UID: "b8c6ff7a-3974-4ca3-9580-4b83b87275e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:55:20.767251 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.767229    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istiod-ca-cert\") pod \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") "
Oct 30 23:55:20.767489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.767262    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-data\") pod \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") "
Oct 30 23:55:20.767489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.767289    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-envoy\") pod \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\" (UID: \"b8c6ff7a-3974-4ca3-9580-4b83b87275e2\") "
Oct 30 23:55:20.767489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.767310    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-workload-certs" pod "b8c6ff7a-3974-4ca3-9580-4b83b87275e2" (UID: "b8c6ff7a-3974-4ca3-9580-4b83b87275e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:55:20.767489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.767315    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-workload-socket" pod "b8c6ff7a-3974-4ca3-9580-4b83b87275e2" (UID: "b8c6ff7a-3974-4ca3-9580-4b83b87275e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:55:20.767770 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.767564    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-data" pod "b8c6ff7a-3974-4ca3-9580-4b83b87275e2" (UID: "b8c6ff7a-3974-4ca3-9580-4b83b87275e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:55:20.767770 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.767682    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istiod-ca-cert" pod "b8c6ff7a-3974-4ca3-9580-4b83b87275e2" (UID: "b8c6ff7a-3974-4ca3-9580-4b83b87275e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:55:20.768057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.767776    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:55:20.768057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.767794    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:55:20.768057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.767807    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:55:20.768057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.767819    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:55:20.768057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.767828    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:55:20.769366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.769323    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-podinfo" pod "b8c6ff7a-3974-4ca3-9580-4b83b87275e2" (UID: "b8c6ff7a-3974-4ca3-9580-4b83b87275e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:55:20.769488 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.769442    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-token" pod "b8c6ff7a-3974-4ca3-9580-4b83b87275e2" (UID: "b8c6ff7a-3974-4ca3-9580-4b83b87275e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:55:20.769488 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.769470    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-kube-api-access-dvdwp" pod "b8c6ff7a-3974-4ca3-9580-4b83b87275e2" (UID: "b8c6ff7a-3974-4ca3-9580-4b83b87275e2"). InnerVolumeSpecName "kube-api-access-dvdwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:55:20.769701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.769664    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-envoy" pod "b8c6ff7a-3974-4ca3-9580-4b83b87275e2" (UID: "b8c6ff7a-3974-4ca3-9580-4b83b87275e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:55:20.868548 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.868520    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvdwp\" (UniqueName: \"kubernetes.io/projected/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-kube-api-access-dvdwp\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:55:20.868548 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.868547    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:55:20.868766 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.868557    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:55:20.868766 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:20.868566    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8c6ff7a-3974-4ca3-9580-4b83b87275e2-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:55:21.541770 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.541730    5078 generic.go:356] "Generic (PLEG): container finished" podID="b8c6ff7a-3974-4ca3-9580-4b83b87275e2" containerID="7766308982c9ee8dfcd04c7a61f171bc05b87efff47aea2db9702f614981a801" exitCode=0
Oct 30 23:55:21.542264 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.541787    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"
Oct 30 23:55:21.542264 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.541812    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z" event={"ID":"b8c6ff7a-3974-4ca3-9580-4b83b87275e2","Type":"ContainerDied","Data":"7766308982c9ee8dfcd04c7a61f171bc05b87efff47aea2db9702f614981a801"}
Oct 30 23:55:21.542264 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.541887    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z" event={"ID":"b8c6ff7a-3974-4ca3-9580-4b83b87275e2","Type":"ContainerDied","Data":"c2d0666be2d56d274ed0dd95b9989d732206cc03d9094899b8005830c3467b65"}
Oct 30 23:55:21.542264 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.541917    5078 scope.go:122] "RemoveContainer" containerID="7766308982c9ee8dfcd04c7a61f171bc05b87efff47aea2db9702f614981a801"
Oct 30 23:55:21.554164 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.554132    5078 scope.go:122] "RemoveContainer" containerID="7766308982c9ee8dfcd04c7a61f171bc05b87efff47aea2db9702f614981a801"
Oct 30 23:55:21.554469 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:55:21.554450    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7766308982c9ee8dfcd04c7a61f171bc05b87efff47aea2db9702f614981a801\": container with ID starting with 7766308982c9ee8dfcd04c7a61f171bc05b87efff47aea2db9702f614981a801 not found: ID does not exist" containerID="7766308982c9ee8dfcd04c7a61f171bc05b87efff47aea2db9702f614981a801"
Oct 30 23:55:21.554533 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.554478    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7766308982c9ee8dfcd04c7a61f171bc05b87efff47aea2db9702f614981a801"} err="failed to get container status \"7766308982c9ee8dfcd04c7a61f171bc05b87efff47aea2db9702f614981a801\": rpc error: code = NotFound desc = could not find container \"7766308982c9ee8dfcd04c7a61f171bc05b87efff47aea2db9702f614981a801\": container with ID starting with 7766308982c9ee8dfcd04c7a61f171bc05b87efff47aea2db9702f614981a801 not found: ID does not exist"
Oct 30 23:55:21.692257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.692230    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn"
Oct 30 23:55:21.711265 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.711239    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"]
Oct 30 23:55:21.760224 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.760200    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-2cfb80-openshift-default-db787f45c-6s52z"]
Oct 30 23:55:21.788012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.786978    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/624ace8a-cc95-410a-b14e-905bbd8e76c9-kube-api-access-29x4t\" (UniqueName: \"kubernetes.io/projected/624ace8a-cc95-410a-b14e-905bbd8e76c9-kube-api-access-29x4t\") pod \"624ace8a-cc95-410a-b14e-905bbd8e76c9\" (UID: \"624ace8a-cc95-410a-b14e-905bbd8e76c9\") "
Oct 30 23:55:21.788012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.787109    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/624ace8a-cc95-410a-b14e-905bbd8e76c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/624ace8a-cc95-410a-b14e-905bbd8e76c9-maas-api-openshift-ca\") pod \"624ace8a-cc95-410a-b14e-905bbd8e76c9\" (UID: \"624ace8a-cc95-410a-b14e-905bbd8e76c9\") "
Oct 30 23:55:21.788012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.787253    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/624ace8a-cc95-410a-b14e-905bbd8e76c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/624ace8a-cc95-410a-b14e-905bbd8e76c9-maas-api-xks-ca\") pod \"624ace8a-cc95-410a-b14e-905bbd8e76c9\" (UID: \"624ace8a-cc95-410a-b14e-905bbd8e76c9\") "
Oct 30 23:55:21.789169 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.789144    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/624ace8a-cc95-410a-b14e-905bbd8e76c9-maas-api-openshift-ca" pod "624ace8a-cc95-410a-b14e-905bbd8e76c9" (UID: "624ace8a-cc95-410a-b14e-905bbd8e76c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:55:21.807151 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.807094    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/624ace8a-cc95-410a-b14e-905bbd8e76c9-maas-api-xks-ca" pod "624ace8a-cc95-410a-b14e-905bbd8e76c9" (UID: "624ace8a-cc95-410a-b14e-905bbd8e76c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:55:21.809394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.809353    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624ace8a-cc95-410a-b14e-905bbd8e76c9-kube-api-access-29x4t" pod "624ace8a-cc95-410a-b14e-905bbd8e76c9" (UID: "624ace8a-cc95-410a-b14e-905bbd8e76c9"). InnerVolumeSpecName "kube-api-access-29x4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:55:21.889200 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.889167    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29x4t\" (UniqueName: \"kubernetes.io/projected/624ace8a-cc95-410a-b14e-905bbd8e76c9-kube-api-access-29x4t\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:55:21.889200 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.889194    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/624ace8a-cc95-410a-b14e-905bbd8e76c9-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:55:21.889200 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:21.889204    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/624ace8a-cc95-410a-b14e-905bbd8e76c9-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:55:22.297856 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:22.297825    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c6ff7a-3974-4ca3-9580-4b83b87275e2" path="/var/lib/kubelet/pods/b8c6ff7a-3974-4ca3-9580-4b83b87275e2/volumes"
Oct 30 23:55:22.550104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:22.550009    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn" event={"ID":"624ace8a-cc95-410a-b14e-905bbd8e76c9","Type":"ContainerDied","Data":"4678ea6127330b44ea193de8cfd1928da4046db3580ea26a300d0cc03ea550c5"}
Oct 30 23:55:22.550104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:22.550065    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4678ea6127330b44ea193de8cfd1928da4046db3580ea26a300d0cc03ea550c5"
Oct 30 23:55:22.550104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:55:22.550065    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn"
Oct 30 23:55:47.590558 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:55:47.590511    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:56:05.066058 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:05.066019    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9"]
Oct 30 23:56:05.114649 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:05.114611    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-71a68a-tv9t9"]
Oct 30 23:56:06.297537 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.297504    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e09f1b6-3e74-408b-8118-e68da0ca5bbb" path="/var/lib/kubelet/pods/1e09f1b6-3e74-408b-8118-e68da0ca5bbb/volumes"
Oct 30 23:56:06.327709 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.327681    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh"]
Oct 30 23:56:06.328126 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.328111    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b8c6ff7a-3974-4ca3-9580-4b83b87275e2" containerName="istio-proxy"
Oct 30 23:56:06.328175 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.328128    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c6ff7a-3974-4ca3-9580-4b83b87275e2" containerName="istio-proxy"
Oct 30 23:56:06.328175 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.328143    5078 cpu_manager.go:414] "containerMap: removing container" podUID="624ace8a-cc95-410a-b14e-905bbd8e76c9" containerName="revoke-keys"
Oct 30 23:56:06.328175 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.328148    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="624ace8a-cc95-410a-b14e-905bbd8e76c9" containerName="revoke-keys"
Oct 30 23:56:06.328281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.328227    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c6ff7a-3974-4ca3-9580-4b83b87275e2" containerName="istio-proxy"
Oct 30 23:56:06.328281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.328238    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="624ace8a-cc95-410a-b14e-905bbd8e76c9" containerName="revoke-keys"
Oct 30 23:56:06.332538 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.332520    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh"
Oct 30 23:56:06.377763 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.377727    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn"]
Oct 30 23:56:06.382362 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.382338    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-15b137-xzjgn"]
Oct 30 23:56:06.384936 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.384919    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:56:06.410379 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.410337    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:56:06.411263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.411235    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvcm4\" (UniqueName: \"kubernetes.io/projected/8d292e2d-e61c-4140-a6e6-fe316625d549-kube-api-access-jvcm4\") pod \"maas-api-revoke-sub-d5d4f234-x8tnh\" (UID: \"8d292e2d-e61c-4140-a6e6-fe316625d549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh"
Oct 30 23:56:06.411408 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.411328    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d292e2d-e61c-4140-a6e6-fe316625d549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5d4f234-x8tnh\" (UID: \"8d292e2d-e61c-4140-a6e6-fe316625d549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh"
Oct 30 23:56:06.411408 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.411363    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d292e2d-e61c-4140-a6e6-fe316625d549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5d4f234-x8tnh\" (UID: \"8d292e2d-e61c-4140-a6e6-fe316625d549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh"
Oct 30 23:56:06.424706 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.424681    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:56:06.424706 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.424683    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:56:06.428528 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.428502    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh"]
Oct 30 23:56:06.512665 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.512627    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d292e2d-e61c-4140-a6e6-fe316625d549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5d4f234-x8tnh\" (UID: \"8d292e2d-e61c-4140-a6e6-fe316625d549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh"
Oct 30 23:56:06.512886 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.512683    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d292e2d-e61c-4140-a6e6-fe316625d549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5d4f234-x8tnh\" (UID: \"8d292e2d-e61c-4140-a6e6-fe316625d549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh"
Oct 30 23:56:06.512886 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.512855    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvcm4\" (UniqueName: \"kubernetes.io/projected/8d292e2d-e61c-4140-a6e6-fe316625d549-kube-api-access-jvcm4\") pod \"maas-api-revoke-sub-d5d4f234-x8tnh\" (UID: \"8d292e2d-e61c-4140-a6e6-fe316625d549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh"
Oct 30 23:56:06.513323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.513299    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d292e2d-e61c-4140-a6e6-fe316625d549-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5d4f234-x8tnh\" (UID: \"8d292e2d-e61c-4140-a6e6-fe316625d549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh"
Oct 30 23:56:06.515112 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.515087    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d292e2d-e61c-4140-a6e6-fe316625d549-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5d4f234-x8tnh\" (UID: \"8d292e2d-e61c-4140-a6e6-fe316625d549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh"
Oct 30 23:56:06.570424 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.570362    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvcm4\" (UniqueName: \"kubernetes.io/projected/8d292e2d-e61c-4140-a6e6-fe316625d549-kube-api-access-jvcm4\") pod \"maas-api-revoke-sub-d5d4f234-x8tnh\" (UID: \"8d292e2d-e61c-4140-a6e6-fe316625d549\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh"
Oct 30 23:56:06.645690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:06.645651    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh"
Oct 30 23:56:07.023945 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:07.023913    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh"]
Oct 30 23:56:07.025491 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:56:07.025464    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d292e2d_e61c_4140_a6e6_fe316625d549.slice/crio-cd3ab4a4a6bc4bd2eefd5fcc42c0c0503aa67e1d15541bf4a3373ac529b35055 WatchSource:0}: Error finding container cd3ab4a4a6bc4bd2eefd5fcc42c0c0503aa67e1d15541bf4a3373ac529b35055: Status 404 returned error can't find the container with id cd3ab4a4a6bc4bd2eefd5fcc42c0c0503aa67e1d15541bf4a3373ac529b35055
Oct 30 23:56:07.802097 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:07.802058    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh" event={"ID":"8d292e2d-e61c-4140-a6e6-fe316625d549","Type":"ContainerStarted","Data":"9a1aa569f3363d7a384380dc6dd9d59b1c81bda575d8a4b23203ee3dc3ee8971"}
Oct 30 23:56:07.802494 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:07.802122    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh" event={"ID":"8d292e2d-e61c-4140-a6e6-fe316625d549","Type":"ContainerStarted","Data":"cd3ab4a4a6bc4bd2eefd5fcc42c0c0503aa67e1d15541bf4a3373ac529b35055"}
Oct 30 23:56:07.969284 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:07.969237    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh" podStartSLOduration=1.9692223439999998 podStartE2EDuration="1.969222344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:56:07.917062696 +0000 UTC m=+2953.804969050" watchObservedRunningTime="2026-10-30 23:56:07.969222344 +0000 UTC m=+2953.857128696"
Oct 30 23:56:08.297987 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:08.297953    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="624ace8a-cc95-410a-b14e-905bbd8e76c9" path="/var/lib/kubelet/pods/624ace8a-cc95-410a-b14e-905bbd8e76c9/volumes"
Oct 30 23:56:08.811042 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:08.811003    5078 generic.go:356] "Generic (PLEG): container finished" podID="8d292e2d-e61c-4140-a6e6-fe316625d549" containerID="9a1aa569f3363d7a384380dc6dd9d59b1c81bda575d8a4b23203ee3dc3ee8971" exitCode=0
Oct 30 23:56:08.811464 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:08.811091    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh" event={"ID":"8d292e2d-e61c-4140-a6e6-fe316625d549","Type":"ContainerDied","Data":"9a1aa569f3363d7a384380dc6dd9d59b1c81bda575d8a4b23203ee3dc3ee8971"}
Oct 30 23:56:09.951916 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:09.951890    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh"
Oct 30 23:56:10.053895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:10.053860    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d292e2d-e61c-4140-a6e6-fe316625d549-kube-api-access-jvcm4\" (UniqueName: \"kubernetes.io/projected/8d292e2d-e61c-4140-a6e6-fe316625d549-kube-api-access-jvcm4\") pod \"8d292e2d-e61c-4140-a6e6-fe316625d549\" (UID: \"8d292e2d-e61c-4140-a6e6-fe316625d549\") "
Oct 30 23:56:10.054117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:10.053996    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d292e2d-e61c-4140-a6e6-fe316625d549-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d292e2d-e61c-4140-a6e6-fe316625d549-maas-api-openshift-ca\") pod \"8d292e2d-e61c-4140-a6e6-fe316625d549\" (UID: \"8d292e2d-e61c-4140-a6e6-fe316625d549\") "
Oct 30 23:56:10.054203 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:10.054119    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d292e2d-e61c-4140-a6e6-fe316625d549-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d292e2d-e61c-4140-a6e6-fe316625d549-maas-api-xks-ca\") pod \"8d292e2d-e61c-4140-a6e6-fe316625d549\" (UID: \"8d292e2d-e61c-4140-a6e6-fe316625d549\") "
Oct 30 23:56:10.054461 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:10.054410    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d292e2d-e61c-4140-a6e6-fe316625d549-maas-api-openshift-ca" pod "8d292e2d-e61c-4140-a6e6-fe316625d549" (UID: "8d292e2d-e61c-4140-a6e6-fe316625d549"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:56:10.054692 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:10.054584    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8d292e2d-e61c-4140-a6e6-fe316625d549-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:10.056447 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:10.056423    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d292e2d-e61c-4140-a6e6-fe316625d549-maas-api-xks-ca" pod "8d292e2d-e61c-4140-a6e6-fe316625d549" (UID: "8d292e2d-e61c-4140-a6e6-fe316625d549"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:56:10.056447 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:10.056439    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d292e2d-e61c-4140-a6e6-fe316625d549-kube-api-access-jvcm4" pod "8d292e2d-e61c-4140-a6e6-fe316625d549" (UID: "8d292e2d-e61c-4140-a6e6-fe316625d549"). InnerVolumeSpecName "kube-api-access-jvcm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:10.156191 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:10.156099    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8d292e2d-e61c-4140-a6e6-fe316625d549-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:10.156191 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:10.156131    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvcm4\" (UniqueName: \"kubernetes.io/projected/8d292e2d-e61c-4140-a6e6-fe316625d549-kube-api-access-jvcm4\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:10.822848 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:10.822814    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh"
Oct 30 23:56:10.823040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:10.822851    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh" event={"ID":"8d292e2d-e61c-4140-a6e6-fe316625d549","Type":"ContainerDied","Data":"cd3ab4a4a6bc4bd2eefd5fcc42c0c0503aa67e1d15541bf4a3373ac529b35055"}
Oct 30 23:56:10.823040 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:10.822903    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd3ab4a4a6bc4bd2eefd5fcc42c0c0503aa67e1d15541bf4a3373ac529b35055"
Oct 30 23:56:12.046443 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:12.046406    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"]
Oct 30 23:56:12.046984 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:12.046729    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469" podUID="abd64869-cfaf-4fec-b73d-90bfb3cc8819" containerName="istio-proxy" containerID="cri-o://3808003d13cdecb9a8f48485a324d9a949afd13f102878e632c3fc058bfd49fe" gracePeriod=30
Oct 30 23:56:12.914060 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:12.914025    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm"]
Oct 30 23:56:12.914506 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:12.914492    5078 cpu_manager.go:414] "containerMap: removing container" podUID="8d292e2d-e61c-4140-a6e6-fe316625d549" containerName="revoke-keys"
Oct 30 23:56:12.914587 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:12.914507    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d292e2d-e61c-4140-a6e6-fe316625d549" containerName="revoke-keys"
Oct 30 23:56:12.914676 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:12.914590    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d292e2d-e61c-4140-a6e6-fe316625d549" containerName="revoke-keys"
Oct 30 23:56:12.917799 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:12.917778    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm"
Oct 30 23:56:13.022510 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:13.022480    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:56:13.022729 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:13.022538    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:56:13.022729 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:13.022539    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:56:13.022729 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:13.022545    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:56:13.024972 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:13.024949    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm"]
Oct 30 23:56:13.088839 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:13.088806    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vff9m\" (UniqueName: \"kubernetes.io/projected/5e39b2c2-f43b-428d-826d-68d2c9f393de-kube-api-access-vff9m\") pod \"maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm\" (UID: \"5e39b2c2-f43b-428d-826d-68d2c9f393de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm"
Oct 30 23:56:13.089314 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:13.088867    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e39b2c2-f43b-428d-826d-68d2c9f393de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm\" (UID: \"5e39b2c2-f43b-428d-826d-68d2c9f393de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm"
Oct 30 23:56:13.089314 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:13.089024    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e39b2c2-f43b-428d-826d-68d2c9f393de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm\" (UID: \"5e39b2c2-f43b-428d-826d-68d2c9f393de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm"
Oct 30 23:56:13.191675 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:13.191261    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e39b2c2-f43b-428d-826d-68d2c9f393de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm\" (UID: \"5e39b2c2-f43b-428d-826d-68d2c9f393de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm"
Oct 30 23:56:13.191675 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:13.191348    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vff9m\" (UniqueName: \"kubernetes.io/projected/5e39b2c2-f43b-428d-826d-68d2c9f393de-kube-api-access-vff9m\") pod \"maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm\" (UID: \"5e39b2c2-f43b-428d-826d-68d2c9f393de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm"
Oct 30 23:56:13.191675 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:13.191442    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e39b2c2-f43b-428d-826d-68d2c9f393de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm\" (UID: \"5e39b2c2-f43b-428d-826d-68d2c9f393de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm"
Oct 30 23:56:13.195620 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:13.192403    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e39b2c2-f43b-428d-826d-68d2c9f393de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm\" (UID: \"5e39b2c2-f43b-428d-826d-68d2c9f393de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm"
Oct 30 23:56:13.212866 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:13.208551    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e39b2c2-f43b-428d-826d-68d2c9f393de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm\" (UID: \"5e39b2c2-f43b-428d-826d-68d2c9f393de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm"
Oct 30 23:56:13.250074 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:13.250042    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vff9m\" (UniqueName: \"kubernetes.io/projected/5e39b2c2-f43b-428d-826d-68d2c9f393de-kube-api-access-vff9m\") pod \"maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm\" (UID: \"5e39b2c2-f43b-428d-826d-68d2c9f393de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm"
Oct 30 23:56:13.530785 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:13.530691    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm"
Oct 30 23:56:13.919008 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:13.918982    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm"]
Oct 30 23:56:13.920993 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:56:13.920966    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e39b2c2_f43b_428d_826d_68d2c9f393de.slice/crio-10644e24f73ec7c7bdffbcb1b545ccf841a0a6b7fbdd60772875669258bf33ab WatchSource:0}: Error finding container 10644e24f73ec7c7bdffbcb1b545ccf841a0a6b7fbdd60772875669258bf33ab: Status 404 returned error can't find the container with id 10644e24f73ec7c7bdffbcb1b545ccf841a0a6b7fbdd60772875669258bf33ab
Oct 30 23:56:14.757331 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:14.757287    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"]
Oct 30 23:56:14.757862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:14.757558    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm" podUID="d6d43311-ddb3-4d7f-901d-025555bc9e37" containerName="istio-proxy" containerID="cri-o://1e852f569dfb880230e338689edd66ec274aace33ed62b0bb7d6e9e8b42dd211" gracePeriod=30
Oct 30 23:56:14.888621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:14.878459    5078 generic.go:356] "Generic (PLEG): container finished" podID="5e39b2c2-f43b-428d-826d-68d2c9f393de" containerID="662b3d9cea8f9ae3e8663bcfde8d665b88242d91583f404a3679f805c9eb46a6" exitCode=0
Oct 30 23:56:14.888621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:14.878531    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm" event={"ID":"5e39b2c2-f43b-428d-826d-68d2c9f393de","Type":"ContainerDied","Data":"662b3d9cea8f9ae3e8663bcfde8d665b88242d91583f404a3679f805c9eb46a6"}
Oct 30 23:56:14.888621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:14.878583    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm" event={"ID":"5e39b2c2-f43b-428d-826d-68d2c9f393de","Type":"ContainerStarted","Data":"10644e24f73ec7c7bdffbcb1b545ccf841a0a6b7fbdd60772875669258bf33ab"}
Oct 30 23:56:15.021584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:15.021500    5078 patch_prober.go:29] interesting pod/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:56:15.021584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:15.021558    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469" podUID="abd64869-cfaf-4fec-b73d-90bfb3cc8819" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:56:15.590420 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:15.590384    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5"]
Oct 30 23:56:15.594078 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:15.594056    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5"
Oct 30 23:56:15.629298 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:15.629261    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2919b211-afd1-4a6a-ab87-6d7a341a610b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5\" (UID: \"2919b211-afd1-4a6a-ab87-6d7a341a610b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5"
Oct 30 23:56:15.629425 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:15.629319    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2919b211-afd1-4a6a-ab87-6d7a341a610b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5\" (UID: \"2919b211-afd1-4a6a-ab87-6d7a341a610b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5"
Oct 30 23:56:15.629503 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:15.629482    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tplt5\" (UniqueName: \"kubernetes.io/projected/2919b211-afd1-4a6a-ab87-6d7a341a610b-kube-api-access-tplt5\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5\" (UID: \"2919b211-afd1-4a6a-ab87-6d7a341a610b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5"
Oct 30 23:56:15.657875 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:15.657846    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5"]
Oct 30 23:56:15.730694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:15.730663    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tplt5\" (UniqueName: \"kubernetes.io/projected/2919b211-afd1-4a6a-ab87-6d7a341a610b-kube-api-access-tplt5\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5\" (UID: \"2919b211-afd1-4a6a-ab87-6d7a341a610b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5"
Oct 30 23:56:15.730913 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:15.730752    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2919b211-afd1-4a6a-ab87-6d7a341a610b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5\" (UID: \"2919b211-afd1-4a6a-ab87-6d7a341a610b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5"
Oct 30 23:56:15.730913 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:15.730796    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2919b211-afd1-4a6a-ab87-6d7a341a610b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5\" (UID: \"2919b211-afd1-4a6a-ab87-6d7a341a610b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5"
Oct 30 23:56:15.731499 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:15.731477    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2919b211-afd1-4a6a-ab87-6d7a341a610b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5\" (UID: \"2919b211-afd1-4a6a-ab87-6d7a341a610b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5"
Oct 30 23:56:15.733183 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:15.733164    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2919b211-afd1-4a6a-ab87-6d7a341a610b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5\" (UID: \"2919b211-afd1-4a6a-ab87-6d7a341a610b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5"
Oct 30 23:56:15.789636 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:15.789591    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tplt5\" (UniqueName: \"kubernetes.io/projected/2919b211-afd1-4a6a-ab87-6d7a341a610b-kube-api-access-tplt5\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5\" (UID: \"2919b211-afd1-4a6a-ab87-6d7a341a610b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5"
Oct 30 23:56:15.907155 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:15.907124    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5"
Oct 30 23:56:16.034394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:16.034369    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm"
Oct 30 23:56:16.134281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:16.134215    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e39b2c2-f43b-428d-826d-68d2c9f393de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e39b2c2-f43b-428d-826d-68d2c9f393de-maas-api-xks-ca\") pod \"5e39b2c2-f43b-428d-826d-68d2c9f393de\" (UID: \"5e39b2c2-f43b-428d-826d-68d2c9f393de\") "
Oct 30 23:56:16.134281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:16.134261    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e39b2c2-f43b-428d-826d-68d2c9f393de-kube-api-access-vff9m\" (UniqueName: \"kubernetes.io/projected/5e39b2c2-f43b-428d-826d-68d2c9f393de-kube-api-access-vff9m\") pod \"5e39b2c2-f43b-428d-826d-68d2c9f393de\" (UID: \"5e39b2c2-f43b-428d-826d-68d2c9f393de\") "
Oct 30 23:56:16.134281 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:16.134282    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e39b2c2-f43b-428d-826d-68d2c9f393de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e39b2c2-f43b-428d-826d-68d2c9f393de-maas-api-openshift-ca\") pod \"5e39b2c2-f43b-428d-826d-68d2c9f393de\" (UID: \"5e39b2c2-f43b-428d-826d-68d2c9f393de\") "
Oct 30 23:56:16.134743 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:16.134718    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e39b2c2-f43b-428d-826d-68d2c9f393de-maas-api-openshift-ca" pod "5e39b2c2-f43b-428d-826d-68d2c9f393de" (UID: "5e39b2c2-f43b-428d-826d-68d2c9f393de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:56:16.136476 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:16.136458    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e39b2c2-f43b-428d-826d-68d2c9f393de-kube-api-access-vff9m" pod "5e39b2c2-f43b-428d-826d-68d2c9f393de" (UID: "5e39b2c2-f43b-428d-826d-68d2c9f393de"). InnerVolumeSpecName "kube-api-access-vff9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:16.136545 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:16.136518    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e39b2c2-f43b-428d-826d-68d2c9f393de-maas-api-xks-ca" pod "5e39b2c2-f43b-428d-826d-68d2c9f393de" (UID: "5e39b2c2-f43b-428d-826d-68d2c9f393de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:56:16.235994 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:16.235961    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e39b2c2-f43b-428d-826d-68d2c9f393de-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:16.235994 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:16.235989    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vff9m\" (UniqueName: \"kubernetes.io/projected/5e39b2c2-f43b-428d-826d-68d2c9f393de-kube-api-access-vff9m\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:16.235994 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:16.235999    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e39b2c2-f43b-428d-826d-68d2c9f393de-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:16.302160 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:16.302129    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5"]
Oct 30 23:56:16.309505 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:56:16.309483    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2919b211_afd1_4a6a_ab87_6d7a341a610b.slice/crio-edede55ada1498c70b4e37dec185fea5ad0d8142a830c8a9dea4783c66e1cdd4 WatchSource:0}: Error finding container edede55ada1498c70b4e37dec185fea5ad0d8142a830c8a9dea4783c66e1cdd4: Status 404 returned error can't find the container with id edede55ada1498c70b4e37dec185fea5ad0d8142a830c8a9dea4783c66e1cdd4
Oct 30 23:56:16.890273 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:16.890247    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm"
Oct 30 23:56:16.890721 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:16.890244    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm" event={"ID":"5e39b2c2-f43b-428d-826d-68d2c9f393de","Type":"ContainerDied","Data":"10644e24f73ec7c7bdffbcb1b545ccf841a0a6b7fbdd60772875669258bf33ab"}
Oct 30 23:56:16.890721 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:16.890369    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10644e24f73ec7c7bdffbcb1b545ccf841a0a6b7fbdd60772875669258bf33ab"
Oct 30 23:56:16.892039 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:16.892014    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5" event={"ID":"2919b211-afd1-4a6a-ab87-6d7a341a610b","Type":"ContainerStarted","Data":"3f10eb8c26319b918bd80b97797eca655d5a28f489a80fd0b9195dd3b662da98"}
Oct 30 23:56:16.892170 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:16.892057    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5" event={"ID":"2919b211-afd1-4a6a-ab87-6d7a341a610b","Type":"ContainerStarted","Data":"edede55ada1498c70b4e37dec185fea5ad0d8142a830c8a9dea4783c66e1cdd4"}
Oct 30 23:56:17.007558 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.007373    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5" podStartSLOduration=2.007355746 podStartE2EDuration="2.007355746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:56:17.007225613 +0000 UTC m=+2962.895131966" watchObservedRunningTime="2026-10-30 23:56:17.007355746 +0000 UTC m=+2962.895262099"
Oct 30 23:56:17.292525 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.292499    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:56:17.348915 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.348887    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-envoy\") pod \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") "
Oct 30 23:56:17.349086 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.348938    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-data\") pod \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") "
Oct 30 23:56:17.349086 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.348959    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-workload-socket\") pod \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") "
Oct 30 23:56:17.349086 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.348998    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-credential-socket\") pod \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") "
Oct 30 23:56:17.349086 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.349016    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-podinfo\") pod \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") "
Oct 30 23:56:17.349086 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.349079    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-token\" (UniqueName: \"kubernetes.io/projected/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-token\") pod \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") "
Oct 30 23:56:17.349374 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.349147    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-workload-certs\") pod \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") "
Oct 30 23:56:17.349374 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.349216    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istiod-ca-cert\") pod \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") "
Oct 30 23:56:17.349374 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.349216    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-data" pod "abd64869-cfaf-4fec-b73d-90bfb3cc8819" (UID: "abd64869-cfaf-4fec-b73d-90bfb3cc8819"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:17.349374 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.349243    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd64869-cfaf-4fec-b73d-90bfb3cc8819-kube-api-access-t44k9\" (UniqueName: \"kubernetes.io/projected/abd64869-cfaf-4fec-b73d-90bfb3cc8819-kube-api-access-t44k9\") pod \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\" (UID: \"abd64869-cfaf-4fec-b73d-90bfb3cc8819\") "
Oct 30 23:56:17.349374 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.349337    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-credential-socket" pod "abd64869-cfaf-4fec-b73d-90bfb3cc8819" (UID: "abd64869-cfaf-4fec-b73d-90bfb3cc8819"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:17.349680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.349353    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-workload-socket" pod "abd64869-cfaf-4fec-b73d-90bfb3cc8819" (UID: "abd64869-cfaf-4fec-b73d-90bfb3cc8819"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:17.349680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.349438    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-workload-certs" pod "abd64869-cfaf-4fec-b73d-90bfb3cc8819" (UID: "abd64869-cfaf-4fec-b73d-90bfb3cc8819"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:17.349795 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.349717    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:17.349795 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.349740    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:17.349795 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.349756    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:17.349795 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.349770    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:17.349795 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.349737    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istiod-ca-cert" pod "abd64869-cfaf-4fec-b73d-90bfb3cc8819" (UID: "abd64869-cfaf-4fec-b73d-90bfb3cc8819"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:56:17.351644 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.351612    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-podinfo" pod "abd64869-cfaf-4fec-b73d-90bfb3cc8819" (UID: "abd64869-cfaf-4fec-b73d-90bfb3cc8819"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:56:17.351759 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.351696    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd64869-cfaf-4fec-b73d-90bfb3cc8819-kube-api-access-t44k9" pod "abd64869-cfaf-4fec-b73d-90bfb3cc8819" (UID: "abd64869-cfaf-4fec-b73d-90bfb3cc8819"). InnerVolumeSpecName "kube-api-access-t44k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:17.351824 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.351807    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-envoy" pod "abd64869-cfaf-4fec-b73d-90bfb3cc8819" (UID: "abd64869-cfaf-4fec-b73d-90bfb3cc8819"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:17.351882 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.351863    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-token" pod "abd64869-cfaf-4fec-b73d-90bfb3cc8819" (UID: "abd64869-cfaf-4fec-b73d-90bfb3cc8819"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:17.450701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.450664    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:17.450701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.450694    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:17.450701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.450704    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:17.450946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.450713    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t44k9\" (UniqueName: \"kubernetes.io/projected/abd64869-cfaf-4fec-b73d-90bfb3cc8819-kube-api-access-t44k9\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:17.450946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.450724    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abd64869-cfaf-4fec-b73d-90bfb3cc8819-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:17.590689 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:56:17.590657    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:56:17.899251 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.899162    5078 generic.go:356] "Generic (PLEG): container finished" podID="abd64869-cfaf-4fec-b73d-90bfb3cc8819" containerID="3808003d13cdecb9a8f48485a324d9a949afd13f102878e632c3fc058bfd49fe" exitCode=0
Oct 30 23:56:17.899251 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.899221    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"
Oct 30 23:56:17.899824 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.899254    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469" event={"ID":"abd64869-cfaf-4fec-b73d-90bfb3cc8819","Type":"ContainerDied","Data":"3808003d13cdecb9a8f48485a324d9a949afd13f102878e632c3fc058bfd49fe"}
Oct 30 23:56:17.899824 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.899313    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469" event={"ID":"abd64869-cfaf-4fec-b73d-90bfb3cc8819","Type":"ContainerDied","Data":"e17f2d25c84439a9da4ca3d0a60a7ae8d6f46beb08f6ca02762e0e05b84b0f38"}
Oct 30 23:56:17.899824 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.899335    5078 scope.go:122] "RemoveContainer" containerID="3808003d13cdecb9a8f48485a324d9a949afd13f102878e632c3fc058bfd49fe"
Oct 30 23:56:17.901464 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.901442    5078 generic.go:356] "Generic (PLEG): container finished" podID="2919b211-afd1-4a6a-ab87-6d7a341a610b" containerID="3f10eb8c26319b918bd80b97797eca655d5a28f489a80fd0b9195dd3b662da98" exitCode=0
Oct 30 23:56:17.901578 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.901494    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5" event={"ID":"2919b211-afd1-4a6a-ab87-6d7a341a610b","Type":"ContainerDied","Data":"3f10eb8c26319b918bd80b97797eca655d5a28f489a80fd0b9195dd3b662da98"}
Oct 30 23:56:17.911559 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.911535    5078 scope.go:122] "RemoveContainer" containerID="3808003d13cdecb9a8f48485a324d9a949afd13f102878e632c3fc058bfd49fe"
Oct 30 23:56:17.911885 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:56:17.911868    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3808003d13cdecb9a8f48485a324d9a949afd13f102878e632c3fc058bfd49fe\": container with ID starting with 3808003d13cdecb9a8f48485a324d9a949afd13f102878e632c3fc058bfd49fe not found: ID does not exist" containerID="3808003d13cdecb9a8f48485a324d9a949afd13f102878e632c3fc058bfd49fe"
Oct 30 23:56:17.911946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:17.911895    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3808003d13cdecb9a8f48485a324d9a949afd13f102878e632c3fc058bfd49fe"} err="failed to get container status \"3808003d13cdecb9a8f48485a324d9a949afd13f102878e632c3fc058bfd49fe\": rpc error: code = NotFound desc = could not find container \"3808003d13cdecb9a8f48485a324d9a949afd13f102878e632c3fc058bfd49fe\": container with ID starting with 3808003d13cdecb9a8f48485a324d9a949afd13f102878e632c3fc058bfd49fe not found: ID does not exist"
Oct 30 23:56:18.065778 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:18.065747    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"]
Oct 30 23:56:18.115562 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:18.115518    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-71a68a-openshift-default-7bf49f766c-6f469"]
Oct 30 23:56:18.298072 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:18.298044    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd64869-cfaf-4fec-b73d-90bfb3cc8819" path="/var/lib/kubelet/pods/abd64869-cfaf-4fec-b73d-90bfb3cc8819/volumes"
Oct 30 23:56:18.386562 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:18.386527    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"]
Oct 30 23:56:19.049430 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.049409    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5"
Oct 30 23:56:19.068497 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.068472    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2919b211-afd1-4a6a-ab87-6d7a341a610b-kube-api-access-tplt5\" (UniqueName: \"kubernetes.io/projected/2919b211-afd1-4a6a-ab87-6d7a341a610b-kube-api-access-tplt5\") pod \"2919b211-afd1-4a6a-ab87-6d7a341a610b\" (UID: \"2919b211-afd1-4a6a-ab87-6d7a341a610b\") "
Oct 30 23:56:19.068662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.068521    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2919b211-afd1-4a6a-ab87-6d7a341a610b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2919b211-afd1-4a6a-ab87-6d7a341a610b-maas-api-xks-ca\") pod \"2919b211-afd1-4a6a-ab87-6d7a341a610b\" (UID: \"2919b211-afd1-4a6a-ab87-6d7a341a610b\") "
Oct 30 23:56:19.068662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.068586    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2919b211-afd1-4a6a-ab87-6d7a341a610b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2919b211-afd1-4a6a-ab87-6d7a341a610b-maas-api-openshift-ca\") pod \"2919b211-afd1-4a6a-ab87-6d7a341a610b\" (UID: \"2919b211-afd1-4a6a-ab87-6d7a341a610b\") "
Oct 30 23:56:19.069005 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.068976    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2919b211-afd1-4a6a-ab87-6d7a341a610b-maas-api-openshift-ca" pod "2919b211-afd1-4a6a-ab87-6d7a341a610b" (UID: "2919b211-afd1-4a6a-ab87-6d7a341a610b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:56:19.070786 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.070765    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2919b211-afd1-4a6a-ab87-6d7a341a610b-maas-api-xks-ca" pod "2919b211-afd1-4a6a-ab87-6d7a341a610b" (UID: "2919b211-afd1-4a6a-ab87-6d7a341a610b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:56:19.070933 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.070911    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2919b211-afd1-4a6a-ab87-6d7a341a610b-kube-api-access-tplt5" pod "2919b211-afd1-4a6a-ab87-6d7a341a610b" (UID: "2919b211-afd1-4a6a-ab87-6d7a341a610b"). InnerVolumeSpecName "kube-api-access-tplt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:19.170502 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.170428    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tplt5\" (UniqueName: \"kubernetes.io/projected/2919b211-afd1-4a6a-ab87-6d7a341a610b-kube-api-access-tplt5\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:19.170502 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.170469    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2919b211-afd1-4a6a-ab87-6d7a341a610b-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:19.170502 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.170480    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2919b211-afd1-4a6a-ab87-6d7a341a610b-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:19.380710 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.380679    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl"]
Oct 30 23:56:19.381087 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.381074    5078 cpu_manager.go:414] "containerMap: removing container" podUID="abd64869-cfaf-4fec-b73d-90bfb3cc8819" containerName="istio-proxy"
Oct 30 23:56:19.381141 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.381088    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd64869-cfaf-4fec-b73d-90bfb3cc8819" containerName="istio-proxy"
Oct 30 23:56:19.381141 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.381106    5078 cpu_manager.go:414] "containerMap: removing container" podUID="5e39b2c2-f43b-428d-826d-68d2c9f393de" containerName="revoke-keys"
Oct 30 23:56:19.381141 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.381111    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e39b2c2-f43b-428d-826d-68d2c9f393de" containerName="revoke-keys"
Oct 30 23:56:19.381141 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.381132    5078 cpu_manager.go:414] "containerMap: removing container" podUID="2919b211-afd1-4a6a-ab87-6d7a341a610b" containerName="revoke-keys"
Oct 30 23:56:19.381141 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.381137    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2919b211-afd1-4a6a-ab87-6d7a341a610b" containerName="revoke-keys"
Oct 30 23:56:19.381333 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.381213    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="2919b211-afd1-4a6a-ab87-6d7a341a610b" containerName="revoke-keys"
Oct 30 23:56:19.381333 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.381223    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd64869-cfaf-4fec-b73d-90bfb3cc8819" containerName="istio-proxy"
Oct 30 23:56:19.381333 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.381231    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e39b2c2-f43b-428d-826d-68d2c9f393de" containerName="revoke-keys"
Oct 30 23:56:19.384425 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.384408    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl"
Oct 30 23:56:19.442319 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.442238    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl"]
Oct 30 23:56:19.474259 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.474225    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx9hm\" (UniqueName: \"kubernetes.io/projected/45326690-de30-4763-9a1e-b9ca57da55bc-kube-api-access-tx9hm\") pod \"maas-api-revoke-sub-926719d0-hwhvl\" (UID: \"45326690-de30-4763-9a1e-b9ca57da55bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl"
Oct 30 23:56:19.474414 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.474289    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45326690-de30-4763-9a1e-b9ca57da55bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-926719d0-hwhvl\" (UID: \"45326690-de30-4763-9a1e-b9ca57da55bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl"
Oct 30 23:56:19.474459 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.474425    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45326690-de30-4763-9a1e-b9ca57da55bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-926719d0-hwhvl\" (UID: \"45326690-de30-4763-9a1e-b9ca57da55bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl"
Oct 30 23:56:19.575791 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.575757    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx9hm\" (UniqueName: \"kubernetes.io/projected/45326690-de30-4763-9a1e-b9ca57da55bc-kube-api-access-tx9hm\") pod \"maas-api-revoke-sub-926719d0-hwhvl\" (UID: \"45326690-de30-4763-9a1e-b9ca57da55bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl"
Oct 30 23:56:19.575991 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.575804    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45326690-de30-4763-9a1e-b9ca57da55bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-926719d0-hwhvl\" (UID: \"45326690-de30-4763-9a1e-b9ca57da55bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl"
Oct 30 23:56:19.575991 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.575834    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45326690-de30-4763-9a1e-b9ca57da55bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-926719d0-hwhvl\" (UID: \"45326690-de30-4763-9a1e-b9ca57da55bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl"
Oct 30 23:56:19.576455 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.576434    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45326690-de30-4763-9a1e-b9ca57da55bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-926719d0-hwhvl\" (UID: \"45326690-de30-4763-9a1e-b9ca57da55bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl"
Oct 30 23:56:19.578445 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.578421    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45326690-de30-4763-9a1e-b9ca57da55bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-926719d0-hwhvl\" (UID: \"45326690-de30-4763-9a1e-b9ca57da55bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl"
Oct 30 23:56:19.634142 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.634115    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx9hm\" (UniqueName: \"kubernetes.io/projected/45326690-de30-4763-9a1e-b9ca57da55bc-kube-api-access-tx9hm\") pod \"maas-api-revoke-sub-926719d0-hwhvl\" (UID: \"45326690-de30-4763-9a1e-b9ca57da55bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl"
Oct 30 23:56:19.698942 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.698864    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl"
Oct 30 23:56:19.916221 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.916193    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5"
Oct 30 23:56:19.916221 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.916207    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5" event={"ID":"2919b211-afd1-4a6a-ab87-6d7a341a610b","Type":"ContainerDied","Data":"edede55ada1498c70b4e37dec185fea5ad0d8142a830c8a9dea4783c66e1cdd4"}
Oct 30 23:56:19.916472 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.916267    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edede55ada1498c70b4e37dec185fea5ad0d8142a830c8a9dea4783c66e1cdd4"
Oct 30 23:56:19.918286 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.918264    5078 generic.go:356] "Generic (PLEG): container finished" podID="d6d43311-ddb3-4d7f-901d-025555bc9e37" containerID="1e852f569dfb880230e338689edd66ec274aace33ed62b0bb7d6e9e8b42dd211" exitCode=0
Oct 30 23:56:19.918425 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.918303    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm" event={"ID":"d6d43311-ddb3-4d7f-901d-025555bc9e37","Type":"ContainerDied","Data":"1e852f569dfb880230e338689edd66ec274aace33ed62b0bb7d6e9e8b42dd211"}
Oct 30 23:56:19.918425 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.918342    5078 scope.go:122] "RemoveContainer" containerID="afb312b3d4f255bee0b333827314ef9ab4f8f0002a021337ca8cd2b0360f6e1a"
Oct 30 23:56:19.994990 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:19.994966    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:56:20.081756 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.081727    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-workload-certs\") pod \"d6d43311-ddb3-4d7f-901d-025555bc9e37\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") "
Oct 30 23:56:20.082114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.081765    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-credential-socket\") pod \"d6d43311-ddb3-4d7f-901d-025555bc9e37\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") "
Oct 30 23:56:20.082114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.081820    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6d43311-ddb3-4d7f-901d-025555bc9e37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d43311-ddb3-4d7f-901d-025555bc9e37-istiod-ca-cert\") pod \"d6d43311-ddb3-4d7f-901d-025555bc9e37\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") "
Oct 30 23:56:20.082114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.081854    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-podinfo\") pod \"d6d43311-ddb3-4d7f-901d-025555bc9e37\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") "
Oct 30 23:56:20.082114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.081965    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-workload-socket\") pod \"d6d43311-ddb3-4d7f-901d-025555bc9e37\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") "
Oct 30 23:56:20.082114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.082011    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-data\") pod \"d6d43311-ddb3-4d7f-901d-025555bc9e37\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") "
Oct 30 23:56:20.082114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.082033    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-workload-certs" pod "d6d43311-ddb3-4d7f-901d-025555bc9e37" (UID: "d6d43311-ddb3-4d7f-901d-025555bc9e37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:20.082114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.082075    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d43311-ddb3-4d7f-901d-025555bc9e37-kube-api-access-k4ttt\" (UniqueName: \"kubernetes.io/projected/d6d43311-ddb3-4d7f-901d-025555bc9e37-kube-api-access-k4ttt\") pod \"d6d43311-ddb3-4d7f-901d-025555bc9e37\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") "
Oct 30 23:56:20.082475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.082122    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-token\" (UniqueName: \"kubernetes.io/projected/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-token\") pod \"d6d43311-ddb3-4d7f-901d-025555bc9e37\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") "
Oct 30 23:56:20.082475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.082175    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-credential-socket" pod "d6d43311-ddb3-4d7f-901d-025555bc9e37" (UID: "d6d43311-ddb3-4d7f-901d-025555bc9e37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:20.082475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.082205    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-workload-socket" pod "d6d43311-ddb3-4d7f-901d-025555bc9e37" (UID: "d6d43311-ddb3-4d7f-901d-025555bc9e37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:20.082475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.082354    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-data" pod "d6d43311-ddb3-4d7f-901d-025555bc9e37" (UID: "d6d43311-ddb3-4d7f-901d-025555bc9e37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:20.082475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.082359    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6d43311-ddb3-4d7f-901d-025555bc9e37-istiod-ca-cert" pod "d6d43311-ddb3-4d7f-901d-025555bc9e37" (UID: "d6d43311-ddb3-4d7f-901d-025555bc9e37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:56:20.082475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.082184    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-envoy\") pod \"d6d43311-ddb3-4d7f-901d-025555bc9e37\" (UID: \"d6d43311-ddb3-4d7f-901d-025555bc9e37\") "
Oct 30 23:56:20.083044 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.083022    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:20.083117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.083051    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:20.083117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.083067    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6d43311-ddb3-4d7f-901d-025555bc9e37-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:20.083117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.083081    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:20.083117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.083097    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:20.084357 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.084332    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-podinfo" pod "d6d43311-ddb3-4d7f-901d-025555bc9e37" (UID: "d6d43311-ddb3-4d7f-901d-025555bc9e37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:56:20.084357 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.084336    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-token" pod "d6d43311-ddb3-4d7f-901d-025555bc9e37" (UID: "d6d43311-ddb3-4d7f-901d-025555bc9e37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:20.084543 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.084532    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-envoy" pod "d6d43311-ddb3-4d7f-901d-025555bc9e37" (UID: "d6d43311-ddb3-4d7f-901d-025555bc9e37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:20.084612 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.084581    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6d43311-ddb3-4d7f-901d-025555bc9e37-kube-api-access-k4ttt" pod "d6d43311-ddb3-4d7f-901d-025555bc9e37" (UID: "d6d43311-ddb3-4d7f-901d-025555bc9e37"). InnerVolumeSpecName "kube-api-access-k4ttt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:20.101068 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.101045    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl"]
Oct 30 23:56:20.116994 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:56:20.116972    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45326690_de30_4763_9a1e_b9ca57da55bc.slice/crio-bce2c837ba6539985cb9180e283e9c9041dbf1b7471d23716d657531c79c65c3 WatchSource:0}: Error finding container bce2c837ba6539985cb9180e283e9c9041dbf1b7471d23716d657531c79c65c3: Status 404 returned error can't find the container with id bce2c837ba6539985cb9180e283e9c9041dbf1b7471d23716d657531c79c65c3
Oct 30 23:56:20.183940 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.183897    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4ttt\" (UniqueName: \"kubernetes.io/projected/d6d43311-ddb3-4d7f-901d-025555bc9e37-kube-api-access-k4ttt\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:20.183940 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.183926    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:20.183940 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.183937    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:20.183940 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.183947    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6d43311-ddb3-4d7f-901d-025555bc9e37-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:20.925462 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.925433    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"
Oct 30 23:56:20.927721 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.925456    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm" event={"ID":"d6d43311-ddb3-4d7f-901d-025555bc9e37","Type":"ContainerDied","Data":"fbbd0dbda3577a75ca38fe9f5e086a23cd4731a3d54c25afcf117ab1891469ee"}
Oct 30 23:56:20.927721 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.925534    5078 scope.go:122] "RemoveContainer" containerID="1e852f569dfb880230e338689edd66ec274aace33ed62b0bb7d6e9e8b42dd211"
Oct 30 23:56:20.927721 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.927327    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl" event={"ID":"45326690-de30-4763-9a1e-b9ca57da55bc","Type":"ContainerStarted","Data":"36585bf59ee991a99fb6cbcafbbf588182a21cf58163d37b5b7950a95e7dac5e"}
Oct 30 23:56:20.927721 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:20.927375    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl" event={"ID":"45326690-de30-4763-9a1e-b9ca57da55bc","Type":"ContainerStarted","Data":"bce2c837ba6539985cb9180e283e9c9041dbf1b7471d23716d657531c79c65c3"}
Oct 30 23:56:21.097900 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:21.097767    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl" podStartSLOduration=2.097749018 podStartE2EDuration="2.097749018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:56:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:56:21.097627797 +0000 UTC m=+2966.985534148" watchObservedRunningTime="2026-10-30 23:56:21.097749018 +0000 UTC m=+2966.985655375"
Oct 30 23:56:21.210307 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:21.210219    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"]
Oct 30 23:56:21.259787 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:21.259758    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-15b137-openshift-default-75bf9bf7cf-t4mxm"]
Oct 30 23:56:21.935878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:21.935843    5078 generic.go:356] "Generic (PLEG): container finished" podID="45326690-de30-4763-9a1e-b9ca57da55bc" containerID="36585bf59ee991a99fb6cbcafbbf588182a21cf58163d37b5b7950a95e7dac5e" exitCode=0
Oct 30 23:56:21.936067 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:21.935933    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl" event={"ID":"45326690-de30-4763-9a1e-b9ca57da55bc","Type":"ContainerDied","Data":"36585bf59ee991a99fb6cbcafbbf588182a21cf58163d37b5b7950a95e7dac5e"}
Oct 30 23:56:22.297058 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:22.297026    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6d43311-ddb3-4d7f-901d-025555bc9e37" path="/var/lib/kubelet/pods/d6d43311-ddb3-4d7f-901d-025555bc9e37/volumes"
Oct 30 23:56:23.084456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:23.084432    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl"
Oct 30 23:56:23.121714 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:23.121677    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45326690-de30-4763-9a1e-b9ca57da55bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45326690-de30-4763-9a1e-b9ca57da55bc-maas-api-openshift-ca\") pod \"45326690-de30-4763-9a1e-b9ca57da55bc\" (UID: \"45326690-de30-4763-9a1e-b9ca57da55bc\") "
Oct 30 23:56:23.121870 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:23.121728    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45326690-de30-4763-9a1e-b9ca57da55bc-kube-api-access-tx9hm\" (UniqueName: \"kubernetes.io/projected/45326690-de30-4763-9a1e-b9ca57da55bc-kube-api-access-tx9hm\") pod \"45326690-de30-4763-9a1e-b9ca57da55bc\" (UID: \"45326690-de30-4763-9a1e-b9ca57da55bc\") "
Oct 30 23:56:23.121870 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:23.121751    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45326690-de30-4763-9a1e-b9ca57da55bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45326690-de30-4763-9a1e-b9ca57da55bc-maas-api-xks-ca\") pod \"45326690-de30-4763-9a1e-b9ca57da55bc\" (UID: \"45326690-de30-4763-9a1e-b9ca57da55bc\") "
Oct 30 23:56:23.122163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:23.122139    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45326690-de30-4763-9a1e-b9ca57da55bc-maas-api-openshift-ca" pod "45326690-de30-4763-9a1e-b9ca57da55bc" (UID: "45326690-de30-4763-9a1e-b9ca57da55bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:56:23.124111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:23.124086    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45326690-de30-4763-9a1e-b9ca57da55bc-kube-api-access-tx9hm" pod "45326690-de30-4763-9a1e-b9ca57da55bc" (UID: "45326690-de30-4763-9a1e-b9ca57da55bc"). InnerVolumeSpecName "kube-api-access-tx9hm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:23.124256 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:23.124127    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45326690-de30-4763-9a1e-b9ca57da55bc-maas-api-xks-ca" pod "45326690-de30-4763-9a1e-b9ca57da55bc" (UID: "45326690-de30-4763-9a1e-b9ca57da55bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:56:23.223397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:23.223310    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45326690-de30-4763-9a1e-b9ca57da55bc-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:23.223397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:23.223338    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx9hm\" (UniqueName: \"kubernetes.io/projected/45326690-de30-4763-9a1e-b9ca57da55bc-kube-api-access-tx9hm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:23.223397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:23.223350    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45326690-de30-4763-9a1e-b9ca57da55bc-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:23.949106 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:23.949078    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl"
Oct 30 23:56:23.949509 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:23.949109    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl" event={"ID":"45326690-de30-4763-9a1e-b9ca57da55bc","Type":"ContainerDied","Data":"bce2c837ba6539985cb9180e283e9c9041dbf1b7471d23716d657531c79c65c3"}
Oct 30 23:56:23.949509 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:23.949158    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bce2c837ba6539985cb9180e283e9c9041dbf1b7471d23716d657531c79c65c3"
Oct 30 23:56:24.061680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:24.061647    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh"]
Oct 30 23:56:24.143521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:24.143489    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6"]
Oct 30 23:56:29.062588 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.062548    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh" podUID="b2da8d7e-5648-494f-8f49-e60fb6060a94" containerName="payload-processing" containerID="cri-o://2b05bb531d9b4f17b152769013cd14e85279496fef2f4dc525fe4630e5f6a7ee" gracePeriod=25
Oct 30 23:56:29.144485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.144431    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6" podUID="79f476c9-ba1a-4abc-8a5e-413b6e1f1e05" containerName="payload-pre-processing" containerID="cri-o://f62ba1d2067dc090f436e98738d155e4195c77213c9f4aea781947e4edf8cec5" gracePeriod=25
Oct 30 23:56:29.413198 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.413172    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh"
Oct 30 23:56:29.437513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.437491    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6"
Oct 30 23:56:29.489748 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.489724    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79f476c9-ba1a-4abc-8a5e-413b6e1f1e05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f476c9-ba1a-4abc-8a5e-413b6e1f1e05-plugins-config-volume\") pod \"79f476c9-ba1a-4abc-8a5e-413b6e1f1e05\" (UID: \"79f476c9-ba1a-4abc-8a5e-413b6e1f1e05\") "
Oct 30 23:56:29.489892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.489814    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2da8d7e-5648-494f-8f49-e60fb6060a94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2da8d7e-5648-494f-8f49-e60fb6060a94-plugins-config-volume\") pod \"b2da8d7e-5648-494f-8f49-e60fb6060a94\" (UID: \"b2da8d7e-5648-494f-8f49-e60fb6060a94\") "
Oct 30 23:56:29.489892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.489843    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2da8d7e-5648-494f-8f49-e60fb6060a94-kube-api-access-p9d7m\" (UniqueName: \"kubernetes.io/projected/b2da8d7e-5648-494f-8f49-e60fb6060a94-kube-api-access-p9d7m\") pod \"b2da8d7e-5648-494f-8f49-e60fb6060a94\" (UID: \"b2da8d7e-5648-494f-8f49-e60fb6060a94\") "
Oct 30 23:56:29.489892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.489883    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79f476c9-ba1a-4abc-8a5e-413b6e1f1e05-kube-api-access-qhzq4\" (UniqueName: \"kubernetes.io/projected/79f476c9-ba1a-4abc-8a5e-413b6e1f1e05-kube-api-access-qhzq4\") pod \"79f476c9-ba1a-4abc-8a5e-413b6e1f1e05\" (UID: \"79f476c9-ba1a-4abc-8a5e-413b6e1f1e05\") "
Oct 30 23:56:29.490173 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.490150    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79f476c9-ba1a-4abc-8a5e-413b6e1f1e05-plugins-config-volume" pod "79f476c9-ba1a-4abc-8a5e-413b6e1f1e05" (UID: "79f476c9-ba1a-4abc-8a5e-413b6e1f1e05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:56:29.490316 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.490176    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2da8d7e-5648-494f-8f49-e60fb6060a94-plugins-config-volume" pod "b2da8d7e-5648-494f-8f49-e60fb6060a94" (UID: "b2da8d7e-5648-494f-8f49-e60fb6060a94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:56:29.492154 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.492130    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2da8d7e-5648-494f-8f49-e60fb6060a94-kube-api-access-p9d7m" pod "b2da8d7e-5648-494f-8f49-e60fb6060a94" (UID: "b2da8d7e-5648-494f-8f49-e60fb6060a94"). InnerVolumeSpecName "kube-api-access-p9d7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:29.492320 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.492300    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79f476c9-ba1a-4abc-8a5e-413b6e1f1e05-kube-api-access-qhzq4" pod "79f476c9-ba1a-4abc-8a5e-413b6e1f1e05" (UID: "79f476c9-ba1a-4abc-8a5e-413b6e1f1e05"). InnerVolumeSpecName "kube-api-access-qhzq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:29.591114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.591085    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79f476c9-ba1a-4abc-8a5e-413b6e1f1e05-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:29.591114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.591113    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2da8d7e-5648-494f-8f49-e60fb6060a94-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:29.591114 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.591121    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9d7m\" (UniqueName: \"kubernetes.io/projected/b2da8d7e-5648-494f-8f49-e60fb6060a94-kube-api-access-p9d7m\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:29.591357 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.591130    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhzq4\" (UniqueName: \"kubernetes.io/projected/79f476c9-ba1a-4abc-8a5e-413b6e1f1e05-kube-api-access-qhzq4\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:29.983311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.983275    5078 generic.go:356] "Generic (PLEG): container finished" podID="b2da8d7e-5648-494f-8f49-e60fb6060a94" containerID="2b05bb531d9b4f17b152769013cd14e85279496fef2f4dc525fe4630e5f6a7ee" exitCode=0
Oct 30 23:56:29.983518 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.983326    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh"
Oct 30 23:56:29.983518 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.983369    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh" event={"ID":"b2da8d7e-5648-494f-8f49-e60fb6060a94","Type":"ContainerDied","Data":"2b05bb531d9b4f17b152769013cd14e85279496fef2f4dc525fe4630e5f6a7ee"}
Oct 30 23:56:29.983518 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.983447    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh" event={"ID":"b2da8d7e-5648-494f-8f49-e60fb6060a94","Type":"ContainerDied","Data":"45e8c2980baae980bcde40b1481dc826c782d840815afe79091c636db659dd42"}
Oct 30 23:56:29.983518 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.983469    5078 scope.go:122] "RemoveContainer" containerID="2b05bb531d9b4f17b152769013cd14e85279496fef2f4dc525fe4630e5f6a7ee"
Oct 30 23:56:29.985204 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.985175    5078 generic.go:356] "Generic (PLEG): container finished" podID="79f476c9-ba1a-4abc-8a5e-413b6e1f1e05" containerID="f62ba1d2067dc090f436e98738d155e4195c77213c9f4aea781947e4edf8cec5" exitCode=0
Oct 30 23:56:29.985363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.985217    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6"
Oct 30 23:56:29.985363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.985227    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6" event={"ID":"79f476c9-ba1a-4abc-8a5e-413b6e1f1e05","Type":"ContainerDied","Data":"f62ba1d2067dc090f436e98738d155e4195c77213c9f4aea781947e4edf8cec5"}
Oct 30 23:56:29.985363 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.985270    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6" event={"ID":"79f476c9-ba1a-4abc-8a5e-413b6e1f1e05","Type":"ContainerDied","Data":"8f2d556cc2a9a9f1af0cb8c92108c8a71e650b857ef7a216dd8f135fddd56762"}
Oct 30 23:56:29.997550 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.995044    5078 scope.go:122] "RemoveContainer" containerID="2b05bb531d9b4f17b152769013cd14e85279496fef2f4dc525fe4630e5f6a7ee"
Oct 30 23:56:29.997894 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:56:29.997868    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b05bb531d9b4f17b152769013cd14e85279496fef2f4dc525fe4630e5f6a7ee\": container with ID starting with 2b05bb531d9b4f17b152769013cd14e85279496fef2f4dc525fe4630e5f6a7ee not found: ID does not exist" containerID="2b05bb531d9b4f17b152769013cd14e85279496fef2f4dc525fe4630e5f6a7ee"
Oct 30 23:56:29.998037 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.997906    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b05bb531d9b4f17b152769013cd14e85279496fef2f4dc525fe4630e5f6a7ee"} err="failed to get container status \"2b05bb531d9b4f17b152769013cd14e85279496fef2f4dc525fe4630e5f6a7ee\": rpc error: code = NotFound desc = could not find container \"2b05bb531d9b4f17b152769013cd14e85279496fef2f4dc525fe4630e5f6a7ee\": container with ID starting with 2b05bb531d9b4f17b152769013cd14e85279496fef2f4dc525fe4630e5f6a7ee not found: ID does not exist"
Oct 30 23:56:29.998037 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:29.997942    5078 scope.go:122] "RemoveContainer" containerID="f62ba1d2067dc090f436e98738d155e4195c77213c9f4aea781947e4edf8cec5"
Oct 30 23:56:30.006256 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:30.006238    5078 scope.go:122] "RemoveContainer" containerID="f62ba1d2067dc090f436e98738d155e4195c77213c9f4aea781947e4edf8cec5"
Oct 30 23:56:30.006500 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:56:30.006484    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f62ba1d2067dc090f436e98738d155e4195c77213c9f4aea781947e4edf8cec5\": container with ID starting with f62ba1d2067dc090f436e98738d155e4195c77213c9f4aea781947e4edf8cec5 not found: ID does not exist" containerID="f62ba1d2067dc090f436e98738d155e4195c77213c9f4aea781947e4edf8cec5"
Oct 30 23:56:30.006547 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:30.006507    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f62ba1d2067dc090f436e98738d155e4195c77213c9f4aea781947e4edf8cec5"} err="failed to get container status \"f62ba1d2067dc090f436e98738d155e4195c77213c9f4aea781947e4edf8cec5\": rpc error: code = NotFound desc = could not find container \"f62ba1d2067dc090f436e98738d155e4195c77213c9f4aea781947e4edf8cec5\": container with ID starting with f62ba1d2067dc090f436e98738d155e4195c77213c9f4aea781947e4edf8cec5 not found: ID does not exist"
Oct 30 23:56:30.148336 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:30.148310    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh"]
Oct 30 23:56:30.197944 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:30.197908    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-a8204d-b68bf65b9-glgjh"]
Oct 30 23:56:30.262428 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:30.262329    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6"]
Oct 30 23:56:30.270038 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:30.270014    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-a8204d-75469964d7-pllz6"]
Oct 30 23:56:30.297476 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:30.297452    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79f476c9-ba1a-4abc-8a5e-413b6e1f1e05" path="/var/lib/kubelet/pods/79f476c9-ba1a-4abc-8a5e-413b6e1f1e05/volumes"
Oct 30 23:56:30.297847 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:30.297834    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2da8d7e-5648-494f-8f49-e60fb6060a94" path="/var/lib/kubelet/pods/b2da8d7e-5648-494f-8f49-e60fb6060a94/volumes"
Oct 30 23:56:31.265690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:31.265657    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5"]
Oct 30 23:56:31.315146 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:31.315115    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-fb14dc-qknx5"]
Oct 30 23:56:32.297915 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:32.297875    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2919b211-afd1-4a6a-ab87-6d7a341a610b" path="/var/lib/kubelet/pods/2919b211-afd1-4a6a-ab87-6d7a341a610b/volumes"
Oct 30 23:56:33.407207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.407124    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg" podUID="85d3f232-2aef-4854-9cec-933ceaaa9752" containerName="main" containerID="cri-o://4131403a409548145f3da42d634f4106b452cbc8c29518e74c55f02816600b69" gracePeriod=45
Oct 30 23:56:33.656973 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.656943    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:56:33.742893 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.742808    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85d3f232-2aef-4854-9cec-933ceaaa9752-kube-api-access-8mr48\" (UniqueName: \"kubernetes.io/projected/85d3f232-2aef-4854-9cec-933ceaaa9752-kube-api-access-8mr48\") pod \"85d3f232-2aef-4854-9cec-933ceaaa9752\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") "
Oct 30 23:56:33.742893 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.742853    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-dshm\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-dshm\") pod \"85d3f232-2aef-4854-9cec-933ceaaa9752\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") "
Oct 30 23:56:33.742893 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.742890    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-model-cache\") pod \"85d3f232-2aef-4854-9cec-933ceaaa9752\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") "
Oct 30 23:56:33.743564 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.742938    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-tmp-dir\") pod \"85d3f232-2aef-4854-9cec-933ceaaa9752\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") "
Oct 30 23:56:33.743564 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.743059    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-home\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-home\") pod \"85d3f232-2aef-4854-9cec-933ceaaa9752\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") "
Oct 30 23:56:33.743564 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.743094    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85d3f232-2aef-4854-9cec-933ceaaa9752-tls-certs\" (UniqueName: \"kubernetes.io/secret/85d3f232-2aef-4854-9cec-933ceaaa9752-tls-certs\") pod \"85d3f232-2aef-4854-9cec-933ceaaa9752\" (UID: \"85d3f232-2aef-4854-9cec-933ceaaa9752\") "
Oct 30 23:56:33.743564 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.743177    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-model-cache" pod "85d3f232-2aef-4854-9cec-933ceaaa9752" (UID: "85d3f232-2aef-4854-9cec-933ceaaa9752"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:33.743564 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.743220    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-tmp-dir" pod "85d3f232-2aef-4854-9cec-933ceaaa9752" (UID: "85d3f232-2aef-4854-9cec-933ceaaa9752"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:33.743564 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.743323    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-home" pod "85d3f232-2aef-4854-9cec-933ceaaa9752" (UID: "85d3f232-2aef-4854-9cec-933ceaaa9752"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:33.743817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.743733    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:33.743817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.743756    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:33.743817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.743771    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:33.745337 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.745311    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-dshm" pod "85d3f232-2aef-4854-9cec-933ceaaa9752" (UID: "85d3f232-2aef-4854-9cec-933ceaaa9752"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:33.745337 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.745333    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85d3f232-2aef-4854-9cec-933ceaaa9752-kube-api-access-8mr48" pod "85d3f232-2aef-4854-9cec-933ceaaa9752" (UID: "85d3f232-2aef-4854-9cec-933ceaaa9752"). InnerVolumeSpecName "kube-api-access-8mr48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:33.745528 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.745398    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85d3f232-2aef-4854-9cec-933ceaaa9752-tls-certs" pod "85d3f232-2aef-4854-9cec-933ceaaa9752" (UID: "85d3f232-2aef-4854-9cec-933ceaaa9752"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:56:33.844558 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.844524    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85d3f232-2aef-4854-9cec-933ceaaa9752-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:33.844733 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.844571    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mr48\" (UniqueName: \"kubernetes.io/projected/85d3f232-2aef-4854-9cec-933ceaaa9752-kube-api-access-8mr48\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:33.844733 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:33.844583    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85d3f232-2aef-4854-9cec-933ceaaa9752-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:34.011985 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:34.011905    5078 generic.go:356] "Generic (PLEG): container finished" podID="85d3f232-2aef-4854-9cec-933ceaaa9752" containerID="4131403a409548145f3da42d634f4106b452cbc8c29518e74c55f02816600b69" exitCode=0
Oct 30 23:56:34.011985 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:34.011972    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"
Oct 30 23:56:34.011985 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:34.011978    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg" event={"ID":"85d3f232-2aef-4854-9cec-933ceaaa9752","Type":"ContainerDied","Data":"4131403a409548145f3da42d634f4106b452cbc8c29518e74c55f02816600b69"}
Oct 30 23:56:34.012220 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:34.012018    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg" event={"ID":"85d3f232-2aef-4854-9cec-933ceaaa9752","Type":"ContainerDied","Data":"bc3951db5debe548ca658f940c5999414527ae6c66135a0592e33175712d885a"}
Oct 30 23:56:34.012220 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:34.012040    5078 scope.go:122] "RemoveContainer" containerID="4131403a409548145f3da42d634f4106b452cbc8c29518e74c55f02816600b69"
Oct 30 23:56:34.021416 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:34.021399    5078 scope.go:122] "RemoveContainer" containerID="4131403a409548145f3da42d634f4106b452cbc8c29518e74c55f02816600b69"
Oct 30 23:56:34.021692 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:56:34.021673    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4131403a409548145f3da42d634f4106b452cbc8c29518e74c55f02816600b69\": container with ID starting with 4131403a409548145f3da42d634f4106b452cbc8c29518e74c55f02816600b69 not found: ID does not exist" containerID="4131403a409548145f3da42d634f4106b452cbc8c29518e74c55f02816600b69"
Oct 30 23:56:34.021749 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:34.021699    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4131403a409548145f3da42d634f4106b452cbc8c29518e74c55f02816600b69"} err="failed to get container status \"4131403a409548145f3da42d634f4106b452cbc8c29518e74c55f02816600b69\": rpc error: code = NotFound desc = could not find container \"4131403a409548145f3da42d634f4106b452cbc8c29518e74c55f02816600b69\": container with ID starting with 4131403a409548145f3da42d634f4106b452cbc8c29518e74c55f02816600b69 not found: ID does not exist"
Oct 30 23:56:34.178653 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:34.178614    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"]
Oct 30 23:56:34.227736 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:34.227703    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-276bec/test-model-276bec-kserve-557fc6f65-ns5vg"]
Oct 30 23:56:34.298659 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:34.298631    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85d3f232-2aef-4854-9cec-933ceaaa9752" path="/var/lib/kubelet/pods/85d3f232-2aef-4854-9cec-933ceaaa9752/volumes"
Oct 30 23:56:35.485557 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:35.485526    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"]
Oct 30 23:56:36.061341 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.061294    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7"]
Oct 30 23:56:36.061797 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.061777    5078 cpu_manager.go:414] "containerMap: removing container" podUID="b2da8d7e-5648-494f-8f49-e60fb6060a94" containerName="payload-processing"
Oct 30 23:56:36.061920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.061798    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2da8d7e-5648-494f-8f49-e60fb6060a94" containerName="payload-processing"
Oct 30 23:56:36.061920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.061817    5078 cpu_manager.go:414] "containerMap: removing container" podUID="85d3f232-2aef-4854-9cec-933ceaaa9752" containerName="main"
Oct 30 23:56:36.061920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.061825    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85d3f232-2aef-4854-9cec-933ceaaa9752" containerName="main"
Oct 30 23:56:36.061920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.061853    5078 cpu_manager.go:414] "containerMap: removing container" podUID="79f476c9-ba1a-4abc-8a5e-413b6e1f1e05" containerName="payload-pre-processing"
Oct 30 23:56:36.061920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.061861    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79f476c9-ba1a-4abc-8a5e-413b6e1f1e05" containerName="payload-pre-processing"
Oct 30 23:56:36.061920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.061888    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d6d43311-ddb3-4d7f-901d-025555bc9e37" containerName="istio-proxy"
Oct 30 23:56:36.061920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.061894    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d43311-ddb3-4d7f-901d-025555bc9e37" containerName="istio-proxy"
Oct 30 23:56:36.061920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.061904    5078 cpu_manager.go:414] "containerMap: removing container" podUID="45326690-de30-4763-9a1e-b9ca57da55bc" containerName="revoke-keys"
Oct 30 23:56:36.061920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.061910    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45326690-de30-4763-9a1e-b9ca57da55bc" containerName="revoke-keys"
Oct 30 23:56:36.061920 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.061920    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d6d43311-ddb3-4d7f-901d-025555bc9e37" containerName="istio-proxy"
Oct 30 23:56:36.062505 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.061927    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6d43311-ddb3-4d7f-901d-025555bc9e37" containerName="istio-proxy"
Oct 30 23:56:36.062505 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.062032    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="85d3f232-2aef-4854-9cec-933ceaaa9752" containerName="main"
Oct 30 23:56:36.062505 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.062048    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="79f476c9-ba1a-4abc-8a5e-413b6e1f1e05" containerName="payload-pre-processing"
Oct 30 23:56:36.062505 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.062060    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d43311-ddb3-4d7f-901d-025555bc9e37" containerName="istio-proxy"
Oct 30 23:56:36.062505 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.062071    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="45326690-de30-4763-9a1e-b9ca57da55bc" containerName="revoke-keys"
Oct 30 23:56:36.062505 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.062080    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6d43311-ddb3-4d7f-901d-025555bc9e37" containerName="istio-proxy"
Oct 30 23:56:36.062505 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.062093    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2da8d7e-5648-494f-8f49-e60fb6060a94" containerName="payload-processing"
Oct 30 23:56:36.066786 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.066767    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7"
Oct 30 23:56:36.128544 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.128523    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:56:36.128694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.128551    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:56:36.172043 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.172009    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a44c52-607f-47a0-b812-f257bd50464b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-276bec-cglg7\" (UID: \"94a44c52-607f-47a0-b812-f257bd50464b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7"
Oct 30 23:56:36.172043 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.172045    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a44c52-607f-47a0-b812-f257bd50464b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-276bec-cglg7\" (UID: \"94a44c52-607f-47a0-b812-f257bd50464b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7"
Oct 30 23:56:36.172251 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.172079    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hx2h\" (UniqueName: \"kubernetes.io/projected/94a44c52-607f-47a0-b812-f257bd50464b-kube-api-access-6hx2h\") pod \"maas-api-revoke-keys-e2e-inf-a-276bec-cglg7\" (UID: \"94a44c52-607f-47a0-b812-f257bd50464b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7"
Oct 30 23:56:36.175814 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.175800    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:56:36.175868 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.175809    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:56:36.178128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.178109    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7"]
Oct 30 23:56:36.273761 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.273726    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a44c52-607f-47a0-b812-f257bd50464b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-276bec-cglg7\" (UID: \"94a44c52-607f-47a0-b812-f257bd50464b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7"
Oct 30 23:56:36.273960 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.273767    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a44c52-607f-47a0-b812-f257bd50464b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-276bec-cglg7\" (UID: \"94a44c52-607f-47a0-b812-f257bd50464b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7"
Oct 30 23:56:36.273960 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.273815    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hx2h\" (UniqueName: \"kubernetes.io/projected/94a44c52-607f-47a0-b812-f257bd50464b-kube-api-access-6hx2h\") pod \"maas-api-revoke-keys-e2e-inf-a-276bec-cglg7\" (UID: \"94a44c52-607f-47a0-b812-f257bd50464b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7"
Oct 30 23:56:36.274435 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.274406    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a44c52-607f-47a0-b812-f257bd50464b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-276bec-cglg7\" (UID: \"94a44c52-607f-47a0-b812-f257bd50464b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7"
Oct 30 23:56:36.276155 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.276138    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a44c52-607f-47a0-b812-f257bd50464b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-276bec-cglg7\" (UID: \"94a44c52-607f-47a0-b812-f257bd50464b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7"
Oct 30 23:56:36.332002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.331938    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hx2h\" (UniqueName: \"kubernetes.io/projected/94a44c52-607f-47a0-b812-f257bd50464b-kube-api-access-6hx2h\") pod \"maas-api-revoke-keys-e2e-inf-a-276bec-cglg7\" (UID: \"94a44c52-607f-47a0-b812-f257bd50464b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7"
Oct 30 23:56:36.379907 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.379872    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7"
Oct 30 23:56:36.759976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:36.759953    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7"]
Oct 30 23:56:36.761707 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:56:36.761679    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94a44c52_607f_47a0_b812_f257bd50464b.slice/crio-cf68046bb71188880fdc08c64b61b7a25761ebf911346ecc2af185cd3f10b43c WatchSource:0}: Error finding container cf68046bb71188880fdc08c64b61b7a25761ebf911346ecc2af185cd3f10b43c: Status 404 returned error can't find the container with id cf68046bb71188880fdc08c64b61b7a25761ebf911346ecc2af185cd3f10b43c
Oct 30 23:56:37.030483 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:37.030403    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7" event={"ID":"94a44c52-607f-47a0-b812-f257bd50464b","Type":"ContainerStarted","Data":"cf68046bb71188880fdc08c64b61b7a25761ebf911346ecc2af185cd3f10b43c"}
Oct 30 23:56:38.069624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:38.063214    5078 generic.go:356] "Generic (PLEG): container finished" podID="94a44c52-607f-47a0-b812-f257bd50464b" containerID="e0b280e75b6f340e12df7d34d838cff313cf5143072cbf42057188bf9e5d5704" exitCode=0
Oct 30 23:56:38.069624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:38.063359    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7" event={"ID":"94a44c52-607f-47a0-b812-f257bd50464b","Type":"ContainerDied","Data":"e0b280e75b6f340e12df7d34d838cff313cf5143072cbf42057188bf9e5d5704"}
Oct 30 23:56:38.891536 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:38.891495    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"]
Oct 30 23:56:38.891743 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:38.891717    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd" podUID="776ad186-9f08-44e9-99e6-f9bc82acc93c" containerName="istio-proxy" containerID="cri-o://5971a2020d2b4ea109b0fab3b87954c6a320e94003f601ce2ae06af3cecbea35" gracePeriod=30
Oct 30 23:56:39.205325 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.205299    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7"
Oct 30 23:56:39.308259 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.308226    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94a44c52-607f-47a0-b812-f257bd50464b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a44c52-607f-47a0-b812-f257bd50464b-maas-api-openshift-ca\") pod \"94a44c52-607f-47a0-b812-f257bd50464b\" (UID: \"94a44c52-607f-47a0-b812-f257bd50464b\") "
Oct 30 23:56:39.308443 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.308319    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94a44c52-607f-47a0-b812-f257bd50464b-kube-api-access-6hx2h\" (UniqueName: \"kubernetes.io/projected/94a44c52-607f-47a0-b812-f257bd50464b-kube-api-access-6hx2h\") pod \"94a44c52-607f-47a0-b812-f257bd50464b\" (UID: \"94a44c52-607f-47a0-b812-f257bd50464b\") "
Oct 30 23:56:39.308443 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.308374    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94a44c52-607f-47a0-b812-f257bd50464b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a44c52-607f-47a0-b812-f257bd50464b-maas-api-xks-ca\") pod \"94a44c52-607f-47a0-b812-f257bd50464b\" (UID: \"94a44c52-607f-47a0-b812-f257bd50464b\") "
Oct 30 23:56:39.308712 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.308686    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94a44c52-607f-47a0-b812-f257bd50464b-maas-api-openshift-ca" pod "94a44c52-607f-47a0-b812-f257bd50464b" (UID: "94a44c52-607f-47a0-b812-f257bd50464b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:56:39.310611 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.310574    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94a44c52-607f-47a0-b812-f257bd50464b-maas-api-xks-ca" pod "94a44c52-607f-47a0-b812-f257bd50464b" (UID: "94a44c52-607f-47a0-b812-f257bd50464b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:56:39.310724 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.310680    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94a44c52-607f-47a0-b812-f257bd50464b-kube-api-access-6hx2h" pod "94a44c52-607f-47a0-b812-f257bd50464b" (UID: "94a44c52-607f-47a0-b812-f257bd50464b"). InnerVolumeSpecName "kube-api-access-6hx2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:39.410128 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.410099    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hx2h\" (UniqueName: \"kubernetes.io/projected/94a44c52-607f-47a0-b812-f257bd50464b-kube-api-access-6hx2h\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:39.410302 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.410139    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94a44c52-607f-47a0-b812-f257bd50464b-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:39.410302 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.410152    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94a44c52-607f-47a0-b812-f257bd50464b-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:39.744403 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.744351    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz"]
Oct 30 23:56:39.744930 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.744911    5078 cpu_manager.go:414] "containerMap: removing container" podUID="94a44c52-607f-47a0-b812-f257bd50464b" containerName="revoke-keys"
Oct 30 23:56:39.744930 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.744932    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94a44c52-607f-47a0-b812-f257bd50464b" containerName="revoke-keys"
Oct 30 23:56:39.745056 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.745007    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="94a44c52-607f-47a0-b812-f257bd50464b" containerName="revoke-keys"
Oct 30 23:56:39.748303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.748286    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz"
Oct 30 23:56:39.809106 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.809065    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz"]
Oct 30 23:56:39.915992 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.915956    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn7zr\" (UniqueName: \"kubernetes.io/projected/d5d72b0f-e820-496a-b570-379f6989f1cc-kube-api-access-wn7zr\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz\" (UID: \"d5d72b0f-e820-496a-b570-379f6989f1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz"
Oct 30 23:56:39.916170 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.916004    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d72b0f-e820-496a-b570-379f6989f1cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz\" (UID: \"d5d72b0f-e820-496a-b570-379f6989f1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz"
Oct 30 23:56:39.916170 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:39.916097    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d72b0f-e820-496a-b570-379f6989f1cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz\" (UID: \"d5d72b0f-e820-496a-b570-379f6989f1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz"
Oct 30 23:56:40.017831 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:40.017738    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn7zr\" (UniqueName: \"kubernetes.io/projected/d5d72b0f-e820-496a-b570-379f6989f1cc-kube-api-access-wn7zr\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz\" (UID: \"d5d72b0f-e820-496a-b570-379f6989f1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz"
Oct 30 23:56:40.017831 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:40.017798    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d72b0f-e820-496a-b570-379f6989f1cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz\" (UID: \"d5d72b0f-e820-496a-b570-379f6989f1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz"
Oct 30 23:56:40.018078 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:40.017835    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d72b0f-e820-496a-b570-379f6989f1cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz\" (UID: \"d5d72b0f-e820-496a-b570-379f6989f1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz"
Oct 30 23:56:40.018455 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:40.018432    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d72b0f-e820-496a-b570-379f6989f1cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz\" (UID: \"d5d72b0f-e820-496a-b570-379f6989f1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz"
Oct 30 23:56:40.020346 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:40.020326    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d72b0f-e820-496a-b570-379f6989f1cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz\" (UID: \"d5d72b0f-e820-496a-b570-379f6989f1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz"
Oct 30 23:56:40.075353 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:40.075325    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7"
Oct 30 23:56:40.075353 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:40.075345    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7" event={"ID":"94a44c52-607f-47a0-b812-f257bd50464b","Type":"ContainerDied","Data":"cf68046bb71188880fdc08c64b61b7a25761ebf911346ecc2af185cd3f10b43c"}
Oct 30 23:56:40.075539 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:40.075395    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf68046bb71188880fdc08c64b61b7a25761ebf911346ecc2af185cd3f10b43c"
Oct 30 23:56:40.079825 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:40.079809    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn7zr\" (UniqueName: \"kubernetes.io/projected/d5d72b0f-e820-496a-b570-379f6989f1cc-kube-api-access-wn7zr\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz\" (UID: \"d5d72b0f-e820-496a-b570-379f6989f1cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz"
Oct 30 23:56:40.360711 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:40.360681    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz"
Oct 30 23:56:40.748072 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:40.748044    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz"]
Oct 30 23:56:40.749675 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:56:40.749642    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5d72b0f_e820_496a_b570_379f6989f1cc.slice/crio-6b908cc3d991383a4a9197e4efaf2a665adc33ff54a4a3f751a0f2c85ab293ff WatchSource:0}: Error finding container 6b908cc3d991383a4a9197e4efaf2a665adc33ff54a4a3f751a0f2c85ab293ff: Status 404 returned error can't find the container with id 6b908cc3d991383a4a9197e4efaf2a665adc33ff54a4a3f751a0f2c85ab293ff
Oct 30 23:56:41.081785 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:41.081753    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz" event={"ID":"d5d72b0f-e820-496a-b570-379f6989f1cc","Type":"ContainerStarted","Data":"15ea60ee66c37f0c1370c89cf5e06e8ae256be2b025ff861f144cc6d87b524c5"}
Oct 30 23:56:41.081931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:41.081808    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz" event={"ID":"d5d72b0f-e820-496a-b570-379f6989f1cc","Type":"ContainerStarted","Data":"6b908cc3d991383a4a9197e4efaf2a665adc33ff54a4a3f751a0f2c85ab293ff"}
Oct 30 23:56:41.198885 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:41.198826    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz" podStartSLOduration=2.19880671 podStartE2EDuration="2.19880671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:56:41.198187024 +0000 UTC m=+2987.086093377" watchObservedRunningTime="2026-10-30 23:56:41.19880671 +0000 UTC m=+2987.086713063"
Oct 30 23:56:42.089415 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:42.089381    5078 generic.go:356] "Generic (PLEG): container finished" podID="d5d72b0f-e820-496a-b570-379f6989f1cc" containerID="15ea60ee66c37f0c1370c89cf5e06e8ae256be2b025ff861f144cc6d87b524c5" exitCode=0
Oct 30 23:56:42.089852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:42.089478    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz" event={"ID":"d5d72b0f-e820-496a-b570-379f6989f1cc","Type":"ContainerDied","Data":"15ea60ee66c37f0c1370c89cf5e06e8ae256be2b025ff861f144cc6d87b524c5"}
Oct 30 23:56:43.229178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:43.229155    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz"
Oct 30 23:56:43.350794 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:43.350725    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5d72b0f-e820-496a-b570-379f6989f1cc-kube-api-access-wn7zr\" (UniqueName: \"kubernetes.io/projected/d5d72b0f-e820-496a-b570-379f6989f1cc-kube-api-access-wn7zr\") pod \"d5d72b0f-e820-496a-b570-379f6989f1cc\" (UID: \"d5d72b0f-e820-496a-b570-379f6989f1cc\") "
Oct 30 23:56:43.350794 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:43.350770    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5d72b0f-e820-496a-b570-379f6989f1cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d72b0f-e820-496a-b570-379f6989f1cc-maas-api-xks-ca\") pod \"d5d72b0f-e820-496a-b570-379f6989f1cc\" (UID: \"d5d72b0f-e820-496a-b570-379f6989f1cc\") "
Oct 30 23:56:43.351021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:43.350843    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5d72b0f-e820-496a-b570-379f6989f1cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d72b0f-e820-496a-b570-379f6989f1cc-maas-api-openshift-ca\") pod \"d5d72b0f-e820-496a-b570-379f6989f1cc\" (UID: \"d5d72b0f-e820-496a-b570-379f6989f1cc\") "
Oct 30 23:56:43.351330 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:43.351301    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5d72b0f-e820-496a-b570-379f6989f1cc-maas-api-openshift-ca" pod "d5d72b0f-e820-496a-b570-379f6989f1cc" (UID: "d5d72b0f-e820-496a-b570-379f6989f1cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:56:43.353136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:43.353105    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5d72b0f-e820-496a-b570-379f6989f1cc-maas-api-xks-ca" pod "d5d72b0f-e820-496a-b570-379f6989f1cc" (UID: "d5d72b0f-e820-496a-b570-379f6989f1cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:56:43.358730 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:43.358704    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5d72b0f-e820-496a-b570-379f6989f1cc-kube-api-access-wn7zr" pod "d5d72b0f-e820-496a-b570-379f6989f1cc" (UID: "d5d72b0f-e820-496a-b570-379f6989f1cc"). InnerVolumeSpecName "kube-api-access-wn7zr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:43.452852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:43.452822    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5d72b0f-e820-496a-b570-379f6989f1cc-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:43.452852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:43.452848    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn7zr\" (UniqueName: \"kubernetes.io/projected/d5d72b0f-e820-496a-b570-379f6989f1cc-kube-api-access-wn7zr\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:43.452852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:43.452858    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5d72b0f-e820-496a-b570-379f6989f1cc-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:44.104761 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.104724    5078 generic.go:356] "Generic (PLEG): container finished" podID="776ad186-9f08-44e9-99e6-f9bc82acc93c" containerID="5971a2020d2b4ea109b0fab3b87954c6a320e94003f601ce2ae06af3cecbea35" exitCode=0
Oct 30 23:56:44.104963 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.104796    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd" event={"ID":"776ad186-9f08-44e9-99e6-f9bc82acc93c","Type":"ContainerDied","Data":"5971a2020d2b4ea109b0fab3b87954c6a320e94003f601ce2ae06af3cecbea35"}
Oct 30 23:56:44.104963 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.104867    5078 scope.go:122] "RemoveContainer" containerID="bf9135ff1b93ab4e7a8f6f0ac5ba38b4e8ae5bc4c48c83566f3ceb0b3810d72a"
Oct 30 23:56:44.106479 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.106457    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz"
Oct 30 23:56:44.106634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.106451    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz" event={"ID":"d5d72b0f-e820-496a-b570-379f6989f1cc","Type":"ContainerDied","Data":"6b908cc3d991383a4a9197e4efaf2a665adc33ff54a4a3f751a0f2c85ab293ff"}
Oct 30 23:56:44.106634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.106611    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b908cc3d991383a4a9197e4efaf2a665adc33ff54a4a3f751a0f2c85ab293ff"
Oct 30 23:56:44.142174 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.142144    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:56:44.260947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.260854    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-workload-socket\") pod \"776ad186-9f08-44e9-99e6-f9bc82acc93c\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") "
Oct 30 23:56:44.260947 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.260914    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-envoy\") pod \"776ad186-9f08-44e9-99e6-f9bc82acc93c\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") "
Oct 30 23:56:44.261501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.260975    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-podinfo\") pod \"776ad186-9f08-44e9-99e6-f9bc82acc93c\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") "
Oct 30 23:56:44.261501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.261025    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776ad186-9f08-44e9-99e6-f9bc82acc93c-kube-api-access-27x9k\" (UniqueName: \"kubernetes.io/projected/776ad186-9f08-44e9-99e6-f9bc82acc93c-kube-api-access-27x9k\") pod \"776ad186-9f08-44e9-99e6-f9bc82acc93c\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") "
Oct 30 23:56:44.261501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.261080    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/776ad186-9f08-44e9-99e6-f9bc82acc93c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776ad186-9f08-44e9-99e6-f9bc82acc93c-istiod-ca-cert\") pod \"776ad186-9f08-44e9-99e6-f9bc82acc93c\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") "
Oct 30 23:56:44.261501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.261147    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-workload-socket" pod "776ad186-9f08-44e9-99e6-f9bc82acc93c" (UID: "776ad186-9f08-44e9-99e6-f9bc82acc93c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:44.261501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.261169    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-token\" (UniqueName: \"kubernetes.io/projected/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-token\") pod \"776ad186-9f08-44e9-99e6-f9bc82acc93c\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") "
Oct 30 23:56:44.261501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.261239    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-data\") pod \"776ad186-9f08-44e9-99e6-f9bc82acc93c\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") "
Oct 30 23:56:44.261501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.261278    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-workload-certs\") pod \"776ad186-9f08-44e9-99e6-f9bc82acc93c\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") "
Oct 30 23:56:44.261501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.261311    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-credential-socket\") pod \"776ad186-9f08-44e9-99e6-f9bc82acc93c\" (UID: \"776ad186-9f08-44e9-99e6-f9bc82acc93c\") "
Oct 30 23:56:44.261501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.261473    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776ad186-9f08-44e9-99e6-f9bc82acc93c-istiod-ca-cert" pod "776ad186-9f08-44e9-99e6-f9bc82acc93c" (UID: "776ad186-9f08-44e9-99e6-f9bc82acc93c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:56:44.262188 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.261666    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-data" pod "776ad186-9f08-44e9-99e6-f9bc82acc93c" (UID: "776ad186-9f08-44e9-99e6-f9bc82acc93c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:44.262188 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.261703    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-credential-socket" pod "776ad186-9f08-44e9-99e6-f9bc82acc93c" (UID: "776ad186-9f08-44e9-99e6-f9bc82acc93c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:44.262188 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.261847    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-workload-certs" pod "776ad186-9f08-44e9-99e6-f9bc82acc93c" (UID: "776ad186-9f08-44e9-99e6-f9bc82acc93c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:44.262188 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.261927    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:44.262188 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.261942    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:44.262188 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.261950    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:44.262188 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.261961    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776ad186-9f08-44e9-99e6-f9bc82acc93c-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:44.263457 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.263433    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-envoy" pod "776ad186-9f08-44e9-99e6-f9bc82acc93c" (UID: "776ad186-9f08-44e9-99e6-f9bc82acc93c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:44.263626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.263505    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776ad186-9f08-44e9-99e6-f9bc82acc93c-kube-api-access-27x9k" pod "776ad186-9f08-44e9-99e6-f9bc82acc93c" (UID: "776ad186-9f08-44e9-99e6-f9bc82acc93c"). InnerVolumeSpecName "kube-api-access-27x9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:44.263626 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.263580    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-podinfo" pod "776ad186-9f08-44e9-99e6-f9bc82acc93c" (UID: "776ad186-9f08-44e9-99e6-f9bc82acc93c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:56:44.263956 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.263939    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-token" pod "776ad186-9f08-44e9-99e6-f9bc82acc93c" (UID: "776ad186-9f08-44e9-99e6-f9bc82acc93c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:44.362915 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.362874    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:44.362915 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.362908    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:44.362915 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.362920    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:44.363173 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.362932    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776ad186-9f08-44e9-99e6-f9bc82acc93c-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:44.363173 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:44.362947    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27x9k\" (UniqueName: \"kubernetes.io/projected/776ad186-9f08-44e9-99e6-f9bc82acc93c-kube-api-access-27x9k\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:45.113152 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:45.113122    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"
Oct 30 23:56:45.113335 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:45.113121    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd" event={"ID":"776ad186-9f08-44e9-99e6-f9bc82acc93c","Type":"ContainerDied","Data":"05b64633bc0689cc2375d6d8d8e98c7c2580d5de905b9f4a6bf21b14b796d688"}
Oct 30 23:56:45.113335 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:45.113249    5078 scope.go:122] "RemoveContainer" containerID="5971a2020d2b4ea109b0fab3b87954c6a320e94003f601ce2ae06af3cecbea35"
Oct 30 23:56:45.328749 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:45.328714    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"]
Oct 30 23:56:45.332353 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:45.332315    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-fb14dc-openshift-default-77c989bfdc-hssqd"]
Oct 30 23:56:46.305164 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:46.305127    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="776ad186-9f08-44e9-99e6-f9bc82acc93c" path="/var/lib/kubelet/pods/776ad186-9f08-44e9-99e6-f9bc82acc93c/volumes"
Oct 30 23:56:47.591555 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:56:47.591511    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:56:47.817301 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:47.817269    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd"]
Oct 30 23:56:50.505958 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.505905    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8" podUID="97878d0b-1dca-46ce-9405-5062573a6ca8" containerName="main" containerID="cri-o://afed14e617b0351f20df9479f7452391138cde54a254ef6daadd4cc78c699afe" gracePeriod=45
Oct 30 23:56:50.760549 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.760486    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:56:50.843222 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.842036    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-home\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-home\") pod \"97878d0b-1dca-46ce-9405-5062573a6ca8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") "
Oct 30 23:56:50.843222 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.842095    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-dshm\") pod \"97878d0b-1dca-46ce-9405-5062573a6ca8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") "
Oct 30 23:56:50.843222 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.842188    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-model-cache\") pod \"97878d0b-1dca-46ce-9405-5062573a6ca8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") "
Oct 30 23:56:50.843222 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.842224    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97878d0b-1dca-46ce-9405-5062573a6ca8-tls-certs\" (UniqueName: \"kubernetes.io/secret/97878d0b-1dca-46ce-9405-5062573a6ca8-tls-certs\") pod \"97878d0b-1dca-46ce-9405-5062573a6ca8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") "
Oct 30 23:56:50.843222 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.842298    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-tmp-dir\") pod \"97878d0b-1dca-46ce-9405-5062573a6ca8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") "
Oct 30 23:56:50.843222 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.842356    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97878d0b-1dca-46ce-9405-5062573a6ca8-kube-api-access-vp8ch\" (UniqueName: \"kubernetes.io/projected/97878d0b-1dca-46ce-9405-5062573a6ca8-kube-api-access-vp8ch\") pod \"97878d0b-1dca-46ce-9405-5062573a6ca8\" (UID: \"97878d0b-1dca-46ce-9405-5062573a6ca8\") "
Oct 30 23:56:50.843222 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.843124    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-home" pod "97878d0b-1dca-46ce-9405-5062573a6ca8" (UID: "97878d0b-1dca-46ce-9405-5062573a6ca8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:50.843818 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.843237    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-tmp-dir" pod "97878d0b-1dca-46ce-9405-5062573a6ca8" (UID: "97878d0b-1dca-46ce-9405-5062573a6ca8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:50.843818 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.843753    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-model-cache" pod "97878d0b-1dca-46ce-9405-5062573a6ca8" (UID: "97878d0b-1dca-46ce-9405-5062573a6ca8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:50.849071 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.849017    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-dshm" pod "97878d0b-1dca-46ce-9405-5062573a6ca8" (UID: "97878d0b-1dca-46ce-9405-5062573a6ca8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:56:50.849071 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.849055    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97878d0b-1dca-46ce-9405-5062573a6ca8-kube-api-access-vp8ch" pod "97878d0b-1dca-46ce-9405-5062573a6ca8" (UID: "97878d0b-1dca-46ce-9405-5062573a6ca8"). InnerVolumeSpecName "kube-api-access-vp8ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:56:50.849292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.849229    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97878d0b-1dca-46ce-9405-5062573a6ca8-tls-certs" pod "97878d0b-1dca-46ce-9405-5062573a6ca8" (UID: "97878d0b-1dca-46ce-9405-5062573a6ca8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:56:50.943375 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.943335    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp8ch\" (UniqueName: \"kubernetes.io/projected/97878d0b-1dca-46ce-9405-5062573a6ca8-kube-api-access-vp8ch\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:50.943375 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.943370    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:50.943375 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.943380    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:50.943672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.943388    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:50.943672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.943398    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97878d0b-1dca-46ce-9405-5062573a6ca8-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:50.943672 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:50.943406    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97878d0b-1dca-46ce-9405-5062573a6ca8-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:56:51.165843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:51.165807    5078 generic.go:356] "Generic (PLEG): container finished" podID="97878d0b-1dca-46ce-9405-5062573a6ca8" containerID="afed14e617b0351f20df9479f7452391138cde54a254ef6daadd4cc78c699afe" exitCode=0
Oct 30 23:56:51.166021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:51.165879    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"
Oct 30 23:56:51.166021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:51.165886    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8" event={"ID":"97878d0b-1dca-46ce-9405-5062573a6ca8","Type":"ContainerDied","Data":"afed14e617b0351f20df9479f7452391138cde54a254ef6daadd4cc78c699afe"}
Oct 30 23:56:51.166021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:51.165917    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8" event={"ID":"97878d0b-1dca-46ce-9405-5062573a6ca8","Type":"ContainerDied","Data":"c7892a3a5427436c6bd2bb0bc7165d72648f2be0d4b6707f2d59505ed00521cf"}
Oct 30 23:56:51.166021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:51.165934    5078 scope.go:122] "RemoveContainer" containerID="afed14e617b0351f20df9479f7452391138cde54a254ef6daadd4cc78c699afe"
Oct 30 23:56:51.175655 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:51.175637    5078 scope.go:122] "RemoveContainer" containerID="afed14e617b0351f20df9479f7452391138cde54a254ef6daadd4cc78c699afe"
Oct 30 23:56:51.175957 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:56:51.175937    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afed14e617b0351f20df9479f7452391138cde54a254ef6daadd4cc78c699afe\": container with ID starting with afed14e617b0351f20df9479f7452391138cde54a254ef6daadd4cc78c699afe not found: ID does not exist" containerID="afed14e617b0351f20df9479f7452391138cde54a254ef6daadd4cc78c699afe"
Oct 30 23:56:51.176006 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:51.175976    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afed14e617b0351f20df9479f7452391138cde54a254ef6daadd4cc78c699afe"} err="failed to get container status \"afed14e617b0351f20df9479f7452391138cde54a254ef6daadd4cc78c699afe\": rpc error: code = NotFound desc = could not find container \"afed14e617b0351f20df9479f7452391138cde54a254ef6daadd4cc78c699afe\": container with ID starting with afed14e617b0351f20df9479f7452391138cde54a254ef6daadd4cc78c699afe not found: ID does not exist"
Oct 30 23:56:51.335150 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:51.335107    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"]
Oct 30 23:56:51.342792 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:51.342766    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-333b59/test-model-333b59-kserve-69887dcb9c-bhhg8"]
Oct 30 23:56:52.297436 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:52.297403    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97878d0b-1dca-46ce-9405-5062573a6ca8" path="/var/lib/kubelet/pods/97878d0b-1dca-46ce-9405-5062573a6ca8/volumes"
Oct 30 23:56:52.818274 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:56:52.818233    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd" podUID="d955624a-a1d8-47f0-88d4-e1e334b21b88" containerName="payload-pre-processing" containerID="cri-o://0a4b1da15b650e0014ab7fe68379066baf41cf5db2f542900e7e541a0175648e" gracePeriod=25
Oct 30 23:57:00.353263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:00.353228    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7"]
Oct 30 23:57:00.401680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:00.401653    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-276bec-cglg7"]
Oct 30 23:57:01.342073 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:01.342034    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd" podUID="d955624a-a1d8-47f0-88d4-e1e334b21b88" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:57:01.568241 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:01.568206    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm"]
Oct 30 23:57:01.576428 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:01.576402    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-a8204d-ncjkm"]
Oct 30 23:57:02.296544 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:02.296510    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e39b2c2-f43b-428d-826d-68d2c9f393de" path="/var/lib/kubelet/pods/5e39b2c2-f43b-428d-826d-68d2c9f393de/volumes"
Oct 30 23:57:02.296959 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:02.296944    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94a44c52-607f-47a0-b812-f257bd50464b" path="/var/lib/kubelet/pods/94a44c52-607f-47a0-b812-f257bd50464b/volumes"
Oct 30 23:57:08.164285 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:08.164248    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz"]
Oct 30 23:57:08.213414 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:08.213376    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-3be34b-5pgsz"]
Oct 30 23:57:08.296954 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:08.296921    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5d72b0f-e820-496a-b570-379f6989f1cc" path="/var/lib/kubelet/pods/d5d72b0f-e820-496a-b570-379f6989f1cc/volumes"
Oct 30 23:57:08.484877 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:08.484802    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"]
Oct 30 23:57:08.485032 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:08.484994    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2" podUID="a61efb0a-769e-4659-9452-3302535611c1" containerName="istio-proxy" containerID="cri-o://c92cd887dce7113ee13a2c54563703d7f9cdbba1fb05ff3491a731e769fde412" gracePeriod=30
Oct 30 23:57:09.322903 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.322870    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt"]
Oct 30 23:57:09.323293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.323222    5078 cpu_manager.go:414] "containerMap: removing container" podUID="776ad186-9f08-44e9-99e6-f9bc82acc93c" containerName="istio-proxy"
Oct 30 23:57:09.323293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.323233    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="776ad186-9f08-44e9-99e6-f9bc82acc93c" containerName="istio-proxy"
Oct 30 23:57:09.323293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.323251    5078 cpu_manager.go:414] "containerMap: removing container" podUID="97878d0b-1dca-46ce-9405-5062573a6ca8" containerName="main"
Oct 30 23:57:09.323293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.323256    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97878d0b-1dca-46ce-9405-5062573a6ca8" containerName="main"
Oct 30 23:57:09.323293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.323272    5078 cpu_manager.go:414] "containerMap: removing container" podUID="776ad186-9f08-44e9-99e6-f9bc82acc93c" containerName="istio-proxy"
Oct 30 23:57:09.323293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.323277    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="776ad186-9f08-44e9-99e6-f9bc82acc93c" containerName="istio-proxy"
Oct 30 23:57:09.323293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.323286    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d5d72b0f-e820-496a-b570-379f6989f1cc" containerName="revoke-keys"
Oct 30 23:57:09.323293 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.323290    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5d72b0f-e820-496a-b570-379f6989f1cc" containerName="revoke-keys"
Oct 30 23:57:09.323567 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.323345    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5d72b0f-e820-496a-b570-379f6989f1cc" containerName="revoke-keys"
Oct 30 23:57:09.323567 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.323357    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="776ad186-9f08-44e9-99e6-f9bc82acc93c" containerName="istio-proxy"
Oct 30 23:57:09.323567 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.323364    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="97878d0b-1dca-46ce-9405-5062573a6ca8" containerName="main"
Oct 30 23:57:09.327978 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.327957    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt"
Oct 30 23:57:09.383102 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.383069    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:57:09.432180 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.432144    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:57:09.432373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.432187    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:57:09.432373 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.432211    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:57:09.434356 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.434330    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt"]
Oct 30 23:57:09.458295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.458270    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-333b59-57mzt\" (UID: \"4f1ae927-6114-44e3-b7c1-2f1ceb19932c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt"
Oct 30 23:57:09.458444 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.458314    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-333b59-57mzt\" (UID: \"4f1ae927-6114-44e3-b7c1-2f1ceb19932c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt"
Oct 30 23:57:09.458508 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.458485    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgvxp\" (UniqueName: \"kubernetes.io/projected/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-kube-api-access-mgvxp\") pod \"maas-api-revoke-keys-e2e-inf-b-333b59-57mzt\" (UID: \"4f1ae927-6114-44e3-b7c1-2f1ceb19932c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt"
Oct 30 23:57:09.560246 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.560217    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-333b59-57mzt\" (UID: \"4f1ae927-6114-44e3-b7c1-2f1ceb19932c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt"
Oct 30 23:57:09.560437 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.560261    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-333b59-57mzt\" (UID: \"4f1ae927-6114-44e3-b7c1-2f1ceb19932c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt"
Oct 30 23:57:09.560437 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.560318    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgvxp\" (UniqueName: \"kubernetes.io/projected/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-kube-api-access-mgvxp\") pod \"maas-api-revoke-keys-e2e-inf-b-333b59-57mzt\" (UID: \"4f1ae927-6114-44e3-b7c1-2f1ceb19932c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt"
Oct 30 23:57:09.560999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.560977    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-333b59-57mzt\" (UID: \"4f1ae927-6114-44e3-b7c1-2f1ceb19932c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt"
Oct 30 23:57:09.562769 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.562754    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-333b59-57mzt\" (UID: \"4f1ae927-6114-44e3-b7c1-2f1ceb19932c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt"
Oct 30 23:57:09.622516 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.622447    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgvxp\" (UniqueName: \"kubernetes.io/projected/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-kube-api-access-mgvxp\") pod \"maas-api-revoke-keys-e2e-inf-b-333b59-57mzt\" (UID: \"4f1ae927-6114-44e3-b7c1-2f1ceb19932c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt"
Oct 30 23:57:09.640272 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:09.640251    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt"
Oct 30 23:57:10.011338 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:10.011314    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt"]
Oct 30 23:57:10.012678 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:57:10.012644    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1ae927_6114_44e3_b7c1_2f1ceb19932c.slice/crio-4f824a97451f6760c8975127ad300f7b7342e246266e44b06d94dbdb210b809c WatchSource:0}: Error finding container 4f824a97451f6760c8975127ad300f7b7342e246266e44b06d94dbdb210b809c: Status 404 returned error can't find the container with id 4f824a97451f6760c8975127ad300f7b7342e246266e44b06d94dbdb210b809c
Oct 30 23:57:10.283168 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:10.283141    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt" event={"ID":"4f1ae927-6114-44e3-b7c1-2f1ceb19932c","Type":"ContainerStarted","Data":"99c6f160c8f14951e06034d2dbde35bf9c7c20166ec33e48323588eaa9c07a2c"}
Oct 30 23:57:10.283292 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:10.283195    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt" event={"ID":"4f1ae927-6114-44e3-b7c1-2f1ceb19932c","Type":"ContainerStarted","Data":"4f824a97451f6760c8975127ad300f7b7342e246266e44b06d94dbdb210b809c"}
Oct 30 23:57:10.375156 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:10.375126    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"]
Oct 30 23:57:10.375572 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:10.375363    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk" podUID="4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" containerName="istio-proxy" containerID="cri-o://c3ad267bca85d209b8890fde01b83ebeed8edc4fd2d82ba3554370e09830f6f5" gracePeriod=30
Oct 30 23:57:10.405612 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:10.405536    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt" podStartSLOduration=1.405524301 podStartE2EDuration="1.405524301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:10.400461044 +0000 UTC m=+3016.288367394" watchObservedRunningTime="2026-10-30 23:57:10.405524301 +0000 UTC m=+3016.293430655"
Oct 30 23:57:11.290394 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:11.290363    5078 generic.go:356] "Generic (PLEG): container finished" podID="4f1ae927-6114-44e3-b7c1-2f1ceb19932c" containerID="99c6f160c8f14951e06034d2dbde35bf9c7c20166ec33e48323588eaa9c07a2c" exitCode=0
Oct 30 23:57:11.290579 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:11.290446    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt" event={"ID":"4f1ae927-6114-44e3-b7c1-2f1ceb19932c","Type":"ContainerDied","Data":"99c6f160c8f14951e06034d2dbde35bf9c7c20166ec33e48323588eaa9c07a2c"}
Oct 30 23:57:11.337477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:11.337440    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd" podUID="d955624a-a1d8-47f0-88d4-e1e334b21b88" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:57:12.426149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:12.426122    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt"
Oct 30 23:57:12.521091 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:12.521061    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-kube-api-access-mgvxp\" (UniqueName: \"kubernetes.io/projected/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-kube-api-access-mgvxp\") pod \"4f1ae927-6114-44e3-b7c1-2f1ceb19932c\" (UID: \"4f1ae927-6114-44e3-b7c1-2f1ceb19932c\") "
Oct 30 23:57:12.521246 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:12.521197    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-maas-api-openshift-ca\") pod \"4f1ae927-6114-44e3-b7c1-2f1ceb19932c\" (UID: \"4f1ae927-6114-44e3-b7c1-2f1ceb19932c\") "
Oct 30 23:57:12.521320 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:12.521260    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-maas-api-xks-ca\") pod \"4f1ae927-6114-44e3-b7c1-2f1ceb19932c\" (UID: \"4f1ae927-6114-44e3-b7c1-2f1ceb19932c\") "
Oct 30 23:57:12.521543 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:12.521523    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-maas-api-openshift-ca" pod "4f1ae927-6114-44e3-b7c1-2f1ceb19932c" (UID: "4f1ae927-6114-44e3-b7c1-2f1ceb19932c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:12.523224 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:12.523205    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-kube-api-access-mgvxp" pod "4f1ae927-6114-44e3-b7c1-2f1ceb19932c" (UID: "4f1ae927-6114-44e3-b7c1-2f1ceb19932c"). InnerVolumeSpecName "kube-api-access-mgvxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:12.523331 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:12.523308    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-maas-api-xks-ca" pod "4f1ae927-6114-44e3-b7c1-2f1ceb19932c" (UID: "4f1ae927-6114-44e3-b7c1-2f1ceb19932c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:57:12.622636 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:12.622526    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:12.622636 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:12.622555    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:12.622636 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:12.622568    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgvxp\" (UniqueName: \"kubernetes.io/projected/4f1ae927-6114-44e3-b7c1-2f1ceb19932c-kube-api-access-mgvxp\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:13.056135 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.056112    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd"
Oct 30 23:57:13.127288 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.127253    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d955624a-a1d8-47f0-88d4-e1e334b21b88-kube-api-access-s64bg\" (UniqueName: \"kubernetes.io/projected/d955624a-a1d8-47f0-88d4-e1e334b21b88-kube-api-access-s64bg\") pod \"d955624a-a1d8-47f0-88d4-e1e334b21b88\" (UID: \"d955624a-a1d8-47f0-88d4-e1e334b21b88\") "
Oct 30 23:57:13.127453 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.127324    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d955624a-a1d8-47f0-88d4-e1e334b21b88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d955624a-a1d8-47f0-88d4-e1e334b21b88-plugins-config-volume\") pod \"d955624a-a1d8-47f0-88d4-e1e334b21b88\" (UID: \"d955624a-a1d8-47f0-88d4-e1e334b21b88\") "
Oct 30 23:57:13.127746 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.127722    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d955624a-a1d8-47f0-88d4-e1e334b21b88-plugins-config-volume" pod "d955624a-a1d8-47f0-88d4-e1e334b21b88" (UID: "d955624a-a1d8-47f0-88d4-e1e334b21b88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:13.129480 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.129462    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d955624a-a1d8-47f0-88d4-e1e334b21b88-kube-api-access-s64bg" pod "d955624a-a1d8-47f0-88d4-e1e334b21b88" (UID: "d955624a-a1d8-47f0-88d4-e1e334b21b88"). InnerVolumeSpecName "kube-api-access-s64bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:13.228315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.228244    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d955624a-a1d8-47f0-88d4-e1e334b21b88-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:13.228315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.228271    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s64bg\" (UniqueName: \"kubernetes.io/projected/d955624a-a1d8-47f0-88d4-e1e334b21b88-kube-api-access-s64bg\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:13.302103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.302072    5078 generic.go:356] "Generic (PLEG): container finished" podID="d955624a-a1d8-47f0-88d4-e1e334b21b88" containerID="0a4b1da15b650e0014ab7fe68379066baf41cf5db2f542900e7e541a0175648e" exitCode=0
Oct 30 23:57:13.302258 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.302111    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd"
Oct 30 23:57:13.302258 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.302156    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd" event={"ID":"d955624a-a1d8-47f0-88d4-e1e334b21b88","Type":"ContainerDied","Data":"0a4b1da15b650e0014ab7fe68379066baf41cf5db2f542900e7e541a0175648e"}
Oct 30 23:57:13.302258 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.302210    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd" event={"ID":"d955624a-a1d8-47f0-88d4-e1e334b21b88","Type":"ContainerDied","Data":"28c302b36706e0f91b344b3493eb33b6c11da672256e6a80204df27406991bdf"}
Oct 30 23:57:13.302258 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.302241    5078 scope.go:122] "RemoveContainer" containerID="0a4b1da15b650e0014ab7fe68379066baf41cf5db2f542900e7e541a0175648e"
Oct 30 23:57:13.303766 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.303744    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt" event={"ID":"4f1ae927-6114-44e3-b7c1-2f1ceb19932c","Type":"ContainerDied","Data":"4f824a97451f6760c8975127ad300f7b7342e246266e44b06d94dbdb210b809c"}
Oct 30 23:57:13.303877 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.303791    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f824a97451f6760c8975127ad300f7b7342e246266e44b06d94dbdb210b809c"
Oct 30 23:57:13.303929 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.303887    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt"
Oct 30 23:57:13.312707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.312689    5078 scope.go:122] "RemoveContainer" containerID="0a4b1da15b650e0014ab7fe68379066baf41cf5db2f542900e7e541a0175648e"
Oct 30 23:57:13.313070 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:57:13.312919    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a4b1da15b650e0014ab7fe68379066baf41cf5db2f542900e7e541a0175648e\": container with ID starting with 0a4b1da15b650e0014ab7fe68379066baf41cf5db2f542900e7e541a0175648e not found: ID does not exist" containerID="0a4b1da15b650e0014ab7fe68379066baf41cf5db2f542900e7e541a0175648e"
Oct 30 23:57:13.313070 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.312954    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4b1da15b650e0014ab7fe68379066baf41cf5db2f542900e7e541a0175648e"} err="failed to get container status \"0a4b1da15b650e0014ab7fe68379066baf41cf5db2f542900e7e541a0175648e\": rpc error: code = NotFound desc = could not find container \"0a4b1da15b650e0014ab7fe68379066baf41cf5db2f542900e7e541a0175648e\": container with ID starting with 0a4b1da15b650e0014ab7fe68379066baf41cf5db2f542900e7e541a0175648e not found: ID does not exist"
Oct 30 23:57:13.530799 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.530773    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd"]
Oct 30 23:57:13.535117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.535096    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-276bec-5dd97d654c-zn5cd"]
Oct 30 23:57:13.737837 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.737809    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:57:13.833389 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.833362    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-workload-socket\") pod \"a61efb0a-769e-4659-9452-3302535611c1\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") "
Oct 30 23:57:13.833516 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.833404    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a61efb0a-769e-4659-9452-3302535611c1-kube-api-access-72gj7\" (UniqueName: \"kubernetes.io/projected/a61efb0a-769e-4659-9452-3302535611c1-kube-api-access-72gj7\") pod \"a61efb0a-769e-4659-9452-3302535611c1\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") "
Oct 30 23:57:13.833516 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.833451    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-istio-data\") pod \"a61efb0a-769e-4659-9452-3302535611c1\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") "
Oct 30 23:57:13.833516 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.833513    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a61efb0a-769e-4659-9452-3302535611c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a61efb0a-769e-4659-9452-3302535611c1-istiod-ca-cert\") pod \"a61efb0a-769e-4659-9452-3302535611c1\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") "
Oct 30 23:57:13.833697 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.833536    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-istio-envoy\") pod \"a61efb0a-769e-4659-9452-3302535611c1\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") "
Oct 30 23:57:13.833697 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.833586    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-credential-socket\") pod \"a61efb0a-769e-4659-9452-3302535611c1\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") "
Oct 30 23:57:13.833697 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.833647    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a61efb0a-769e-4659-9452-3302535611c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a61efb0a-769e-4659-9452-3302535611c1-istio-podinfo\") pod \"a61efb0a-769e-4659-9452-3302535611c1\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") "
Oct 30 23:57:13.833697 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.833650    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-workload-socket" pod "a61efb0a-769e-4659-9452-3302535611c1" (UID: "a61efb0a-769e-4659-9452-3302535611c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:13.833697 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.833684    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-workload-certs\") pod \"a61efb0a-769e-4659-9452-3302535611c1\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") "
Oct 30 23:57:13.834023 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.833728    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a61efb0a-769e-4659-9452-3302535611c1-istio-token\" (UniqueName: \"kubernetes.io/projected/a61efb0a-769e-4659-9452-3302535611c1-istio-token\") pod \"a61efb0a-769e-4659-9452-3302535611c1\" (UID: \"a61efb0a-769e-4659-9452-3302535611c1\") "
Oct 30 23:57:13.834023 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.834001    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-credential-socket" pod "a61efb0a-769e-4659-9452-3302535611c1" (UID: "a61efb0a-769e-4659-9452-3302535611c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:13.834154 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.834118    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-workload-certs" pod "a61efb0a-769e-4659-9452-3302535611c1" (UID: "a61efb0a-769e-4659-9452-3302535611c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:13.834209 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.834129    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:13.834209 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.834156    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61efb0a-769e-4659-9452-3302535611c1-istiod-ca-cert" pod "a61efb0a-769e-4659-9452-3302535611c1" (UID: "a61efb0a-769e-4659-9452-3302535611c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:13.834209 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.834184    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:13.834209 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.834189    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-istio-data" pod "a61efb0a-769e-4659-9452-3302535611c1" (UID: "a61efb0a-769e-4659-9452-3302535611c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:13.835944 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.835916    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61efb0a-769e-4659-9452-3302535611c1-kube-api-access-72gj7" pod "a61efb0a-769e-4659-9452-3302535611c1" (UID: "a61efb0a-769e-4659-9452-3302535611c1"). InnerVolumeSpecName "kube-api-access-72gj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:13.835944 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.835923    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-istio-envoy" pod "a61efb0a-769e-4659-9452-3302535611c1" (UID: "a61efb0a-769e-4659-9452-3302535611c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:13.836074 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.836015    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a61efb0a-769e-4659-9452-3302535611c1-istio-podinfo" pod "a61efb0a-769e-4659-9452-3302535611c1" (UID: "a61efb0a-769e-4659-9452-3302535611c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:57:13.836074 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.836056    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61efb0a-769e-4659-9452-3302535611c1-istio-token" pod "a61efb0a-769e-4659-9452-3302535611c1" (UID: "a61efb0a-769e-4659-9452-3302535611c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:13.935048 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.935016    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a61efb0a-769e-4659-9452-3302535611c1-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:13.935048 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.935044    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:13.935048 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.935054    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a61efb0a-769e-4659-9452-3302535611c1-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:13.935351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.935062    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72gj7\" (UniqueName: \"kubernetes.io/projected/a61efb0a-769e-4659-9452-3302535611c1-kube-api-access-72gj7\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:13.935351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.935073    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:13.935351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.935081    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a61efb0a-769e-4659-9452-3302535611c1-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:13.935351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:13.935090    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a61efb0a-769e-4659-9452-3302535611c1-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:14.296862 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:14.296832    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d955624a-a1d8-47f0-88d4-e1e334b21b88" path="/var/lib/kubelet/pods/d955624a-a1d8-47f0-88d4-e1e334b21b88/volumes"
Oct 30 23:57:14.310119 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:14.310095    5078 generic.go:356] "Generic (PLEG): container finished" podID="a61efb0a-769e-4659-9452-3302535611c1" containerID="c92cd887dce7113ee13a2c54563703d7f9cdbba1fb05ff3491a731e769fde412" exitCode=0
Oct 30 23:57:14.310264 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:14.310140    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"
Oct 30 23:57:14.310264 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:14.310173    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2" event={"ID":"a61efb0a-769e-4659-9452-3302535611c1","Type":"ContainerDied","Data":"c92cd887dce7113ee13a2c54563703d7f9cdbba1fb05ff3491a731e769fde412"}
Oct 30 23:57:14.310264 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:14.310228    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2" event={"ID":"a61efb0a-769e-4659-9452-3302535611c1","Type":"ContainerDied","Data":"96e20c5d7530d763ccd745d82990c64b0e86645a2e83d17fae9a1a5b4b122f36"}
Oct 30 23:57:14.310264 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:14.310252    5078 scope.go:122] "RemoveContainer" containerID="c92cd887dce7113ee13a2c54563703d7f9cdbba1fb05ff3491a731e769fde412"
Oct 30 23:57:14.319260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:14.319244    5078 scope.go:122] "RemoveContainer" containerID="7c30fd86464c469f0459151ddab1ff781b4e8f12d15a68a641635c68eaf34e9e"
Oct 30 23:57:14.327325 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:14.327297    5078 scope.go:122] "RemoveContainer" containerID="c92cd887dce7113ee13a2c54563703d7f9cdbba1fb05ff3491a731e769fde412"
Oct 30 23:57:14.327586 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:57:14.327570    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c92cd887dce7113ee13a2c54563703d7f9cdbba1fb05ff3491a731e769fde412\": container with ID starting with c92cd887dce7113ee13a2c54563703d7f9cdbba1fb05ff3491a731e769fde412 not found: ID does not exist" containerID="c92cd887dce7113ee13a2c54563703d7f9cdbba1fb05ff3491a731e769fde412"
Oct 30 23:57:14.327660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:14.327607    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c92cd887dce7113ee13a2c54563703d7f9cdbba1fb05ff3491a731e769fde412"} err="failed to get container status \"c92cd887dce7113ee13a2c54563703d7f9cdbba1fb05ff3491a731e769fde412\": rpc error: code = NotFound desc = could not find container \"c92cd887dce7113ee13a2c54563703d7f9cdbba1fb05ff3491a731e769fde412\": container with ID starting with c92cd887dce7113ee13a2c54563703d7f9cdbba1fb05ff3491a731e769fde412 not found: ID does not exist"
Oct 30 23:57:14.327660 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:14.327630    5078 scope.go:122] "RemoveContainer" containerID="7c30fd86464c469f0459151ddab1ff781b4e8f12d15a68a641635c68eaf34e9e"
Oct 30 23:57:14.327887 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:57:14.327867    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c30fd86464c469f0459151ddab1ff781b4e8f12d15a68a641635c68eaf34e9e\": container with ID starting with 7c30fd86464c469f0459151ddab1ff781b4e8f12d15a68a641635c68eaf34e9e not found: ID does not exist" containerID="7c30fd86464c469f0459151ddab1ff781b4e8f12d15a68a641635c68eaf34e9e"
Oct 30 23:57:14.327946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:14.327893    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c30fd86464c469f0459151ddab1ff781b4e8f12d15a68a641635c68eaf34e9e"} err="failed to get container status \"7c30fd86464c469f0459151ddab1ff781b4e8f12d15a68a641635c68eaf34e9e\": rpc error: code = NotFound desc = could not find container \"7c30fd86464c469f0459151ddab1ff781b4e8f12d15a68a641635c68eaf34e9e\": container with ID starting with 7c30fd86464c469f0459151ddab1ff781b4e8f12d15a68a641635c68eaf34e9e not found: ID does not exist"
Oct 30 23:57:14.479513 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:14.479478    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"]
Oct 30 23:57:14.529258 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:14.529227    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-276bec-openshift-default-65cfdf549-xmzr2"]
Oct 30 23:57:14.586776 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:14.586698    5078 patch_prober.go:29] interesting pod/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:57:14.586776 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:14.586753    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk" podUID="4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:57:15.366662 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.366628    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"]
Oct 30 23:57:15.366846 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.366809    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c" podUID="c8b4ee1d-90dd-416a-9955-cc54ad064723" containerName="istio-proxy" containerID="cri-o://2d792d082bc075db64d00d86bd328798a1a312e75575061d5a9f48639fc4681c" gracePeriod=30
Oct 30 23:57:15.650642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.650589    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:57:15.753577 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.753546    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-envoy\") pod \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") "
Oct 30 23:57:15.753754 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.753619    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-workload-socket\") pod \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") "
Oct 30 23:57:15.753754 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.753651    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-podinfo\") pod \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") "
Oct 30 23:57:15.753754 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.753685    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istiod-ca-cert\") pod \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") "
Oct 30 23:57:15.753935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.753754    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-credential-socket\") pod \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") "
Oct 30 23:57:15.753935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.753866    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-kube-api-access-v52bd\" (UniqueName: \"kubernetes.io/projected/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-kube-api-access-v52bd\") pod \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") "
Oct 30 23:57:15.753935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.753913    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-workload-socket" pod "4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" (UID: "4e7542f5-5c45-4cf1-b0b0-0f079709f0d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:15.754092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.753940    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-data\") pod \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") "
Oct 30 23:57:15.754092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.754023    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-token\" (UniqueName: \"kubernetes.io/projected/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-token\") pod \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") "
Oct 30 23:57:15.754092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.754038    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-credential-socket" pod "4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" (UID: "4e7542f5-5c45-4cf1-b0b0-0f079709f0d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:15.754092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.754058    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-workload-certs\") pod \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\" (UID: \"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4\") "
Oct 30 23:57:15.754349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.754131    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istiod-ca-cert" pod "4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" (UID: "4e7542f5-5c45-4cf1-b0b0-0f079709f0d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:15.754412 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.754394    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:15.754466 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.754416    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:15.754466 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.754432    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:15.754723 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.754636    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-workload-certs" pod "4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" (UID: "4e7542f5-5c45-4cf1-b0b0-0f079709f0d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:15.754938 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.754779    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-data" pod "4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" (UID: "4e7542f5-5c45-4cf1-b0b0-0f079709f0d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:15.755939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.755908    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-envoy" pod "4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" (UID: "4e7542f5-5c45-4cf1-b0b0-0f079709f0d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:15.756046 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.756028    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-kube-api-access-v52bd" pod "4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" (UID: "4e7542f5-5c45-4cf1-b0b0-0f079709f0d4"). InnerVolumeSpecName "kube-api-access-v52bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:15.756101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.756076    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-podinfo" pod "4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" (UID: "4e7542f5-5c45-4cf1-b0b0-0f079709f0d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:57:15.756239 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.756225    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-token" pod "4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" (UID: "4e7542f5-5c45-4cf1-b0b0-0f079709f0d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:15.855834 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.855806    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v52bd\" (UniqueName: \"kubernetes.io/projected/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-kube-api-access-v52bd\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:15.855834 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.855832    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:15.855834 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.855842    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:15.856064 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.855851    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:15.856064 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.855860    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:15.856064 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.855868    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:15.908138 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.908066    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"]
Oct 30 23:57:15.923912 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:15.923882    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"]
Oct 30 23:57:16.296268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.296238    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a61efb0a-769e-4659-9452-3302535611c1" path="/var/lib/kubelet/pods/a61efb0a-769e-4659-9452-3302535611c1/volumes"
Oct 30 23:57:16.324295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.324269    5078 generic.go:356] "Generic (PLEG): container finished" podID="4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" containerID="c3ad267bca85d209b8890fde01b83ebeed8edc4fd2d82ba3554370e09830f6f5" exitCode=0
Oct 30 23:57:16.324465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.324335    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"
Oct 30 23:57:16.324465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.324353    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk" event={"ID":"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4","Type":"ContainerDied","Data":"c3ad267bca85d209b8890fde01b83ebeed8edc4fd2d82ba3554370e09830f6f5"}
Oct 30 23:57:16.324465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.324412    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk" event={"ID":"4e7542f5-5c45-4cf1-b0b0-0f079709f0d4","Type":"ContainerDied","Data":"4a7d0b5931ecc07e90fa4eb8a32d7de4271226df4fa53ce5989d5dc02c3a7c1d"}
Oct 30 23:57:16.324465 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.324434    5078 scope.go:122] "RemoveContainer" containerID="c3ad267bca85d209b8890fde01b83ebeed8edc4fd2d82ba3554370e09830f6f5"
Oct 30 23:57:16.333366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.333347    5078 scope.go:122] "RemoveContainer" containerID="eb96f3c549197e968533ae14f3252a52b9ecbae016de3e8e1818b3faef46ea93"
Oct 30 23:57:16.341734 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.341718    5078 scope.go:122] "RemoveContainer" containerID="c3ad267bca85d209b8890fde01b83ebeed8edc4fd2d82ba3554370e09830f6f5"
Oct 30 23:57:16.341990 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:57:16.341973    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3ad267bca85d209b8890fde01b83ebeed8edc4fd2d82ba3554370e09830f6f5\": container with ID starting with c3ad267bca85d209b8890fde01b83ebeed8edc4fd2d82ba3554370e09830f6f5 not found: ID does not exist" containerID="c3ad267bca85d209b8890fde01b83ebeed8edc4fd2d82ba3554370e09830f6f5"
Oct 30 23:57:16.342030 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.341996    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3ad267bca85d209b8890fde01b83ebeed8edc4fd2d82ba3554370e09830f6f5"} err="failed to get container status \"c3ad267bca85d209b8890fde01b83ebeed8edc4fd2d82ba3554370e09830f6f5\": rpc error: code = NotFound desc = could not find container \"c3ad267bca85d209b8890fde01b83ebeed8edc4fd2d82ba3554370e09830f6f5\": container with ID starting with c3ad267bca85d209b8890fde01b83ebeed8edc4fd2d82ba3554370e09830f6f5 not found: ID does not exist"
Oct 30 23:57:16.342030 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.342012    5078 scope.go:122] "RemoveContainer" containerID="eb96f3c549197e968533ae14f3252a52b9ecbae016de3e8e1818b3faef46ea93"
Oct 30 23:57:16.342219 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:57:16.342204    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb96f3c549197e968533ae14f3252a52b9ecbae016de3e8e1818b3faef46ea93\": container with ID starting with eb96f3c549197e968533ae14f3252a52b9ecbae016de3e8e1818b3faef46ea93 not found: ID does not exist" containerID="eb96f3c549197e968533ae14f3252a52b9ecbae016de3e8e1818b3faef46ea93"
Oct 30 23:57:16.342264 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.342223    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb96f3c549197e968533ae14f3252a52b9ecbae016de3e8e1818b3faef46ea93"} err="failed to get container status \"eb96f3c549197e968533ae14f3252a52b9ecbae016de3e8e1818b3faef46ea93\": rpc error: code = NotFound desc = could not find container \"eb96f3c549197e968533ae14f3252a52b9ecbae016de3e8e1818b3faef46ea93\": container with ID starting with eb96f3c549197e968533ae14f3252a52b9ecbae016de3e8e1818b3faef46ea93 not found: ID does not exist"
Oct 30 23:57:16.402584 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.402554    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt"]
Oct 30 23:57:16.403151 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.403132    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" containerName="istio-proxy"
Oct 30 23:57:16.403205 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.403156    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" containerName="istio-proxy"
Oct 30 23:57:16.403205 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.403172    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" containerName="istio-proxy"
Oct 30 23:57:16.403205 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.403180    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" containerName="istio-proxy"
Oct 30 23:57:16.403205 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.403189    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a61efb0a-769e-4659-9452-3302535611c1" containerName="istio-proxy"
Oct 30 23:57:16.403205 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.403197    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a61efb0a-769e-4659-9452-3302535611c1" containerName="istio-proxy"
Oct 30 23:57:16.403360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.403228    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d955624a-a1d8-47f0-88d4-e1e334b21b88" containerName="payload-pre-processing"
Oct 30 23:57:16.403360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.403236    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d955624a-a1d8-47f0-88d4-e1e334b21b88" containerName="payload-pre-processing"
Oct 30 23:57:16.403360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.403261    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4f1ae927-6114-44e3-b7c1-2f1ceb19932c" containerName="revoke-keys"
Oct 30 23:57:16.403360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.403269    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f1ae927-6114-44e3-b7c1-2f1ceb19932c" containerName="revoke-keys"
Oct 30 23:57:16.403360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.403281    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a61efb0a-769e-4659-9452-3302535611c1" containerName="istio-proxy"
Oct 30 23:57:16.403360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.403289    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a61efb0a-769e-4659-9452-3302535611c1" containerName="istio-proxy"
Oct 30 23:57:16.403555 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.403388    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" containerName="istio-proxy"
Oct 30 23:57:16.403555 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.403398    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" containerName="istio-proxy"
Oct 30 23:57:16.403555 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.403410    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a61efb0a-769e-4659-9452-3302535611c1" containerName="istio-proxy"
Oct 30 23:57:16.403555 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.403423    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="776ad186-9f08-44e9-99e6-f9bc82acc93c" containerName="istio-proxy"
Oct 30 23:57:16.403555 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.403434    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d955624a-a1d8-47f0-88d4-e1e334b21b88" containerName="payload-pre-processing"
Oct 30 23:57:16.403555 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.403449    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f1ae927-6114-44e3-b7c1-2f1ceb19932c" containerName="revoke-keys"
Oct 30 23:57:16.408501 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.408482    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt"
Oct 30 23:57:16.460472 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.460439    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spkxj\" (UniqueName: \"kubernetes.io/projected/a7f1998f-5d7a-493b-bf98-30de70ac25b7-kube-api-access-spkxj\") pod \"maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt\" (UID: \"a7f1998f-5d7a-493b-bf98-30de70ac25b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt"
Oct 30 23:57:16.460644 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.460494    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f1998f-5d7a-493b-bf98-30de70ac25b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt\" (UID: \"a7f1998f-5d7a-493b-bf98-30de70ac25b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt"
Oct 30 23:57:16.460735 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.460714    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f1998f-5d7a-493b-bf98-30de70ac25b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt\" (UID: \"a7f1998f-5d7a-493b-bf98-30de70ac25b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt"
Oct 30 23:57:16.463311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.463288    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:57:16.463311 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.463303    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:57:16.466247 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.466218    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt"]
Oct 30 23:57:16.482879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.482847    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:57:16.482879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.482851    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:57:16.491635 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.491610    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"]
Oct 30 23:57:16.514833 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.514808    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a8204d-openshift-default-8cc557fbc-5lmxk"]
Oct 30 23:57:16.561642 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.561571    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f1998f-5d7a-493b-bf98-30de70ac25b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt\" (UID: \"a7f1998f-5d7a-493b-bf98-30de70ac25b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt"
Oct 30 23:57:16.561731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.561649    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spkxj\" (UniqueName: \"kubernetes.io/projected/a7f1998f-5d7a-493b-bf98-30de70ac25b7-kube-api-access-spkxj\") pod \"maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt\" (UID: \"a7f1998f-5d7a-493b-bf98-30de70ac25b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt"
Oct 30 23:57:16.561731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.561672    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f1998f-5d7a-493b-bf98-30de70ac25b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt\" (UID: \"a7f1998f-5d7a-493b-bf98-30de70ac25b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt"
Oct 30 23:57:16.562275 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.562258    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f1998f-5d7a-493b-bf98-30de70ac25b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt\" (UID: \"a7f1998f-5d7a-493b-bf98-30de70ac25b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt"
Oct 30 23:57:16.564057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.564041    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f1998f-5d7a-493b-bf98-30de70ac25b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt\" (UID: \"a7f1998f-5d7a-493b-bf98-30de70ac25b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt"
Oct 30 23:57:16.621323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.621295    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spkxj\" (UniqueName: \"kubernetes.io/projected/a7f1998f-5d7a-493b-bf98-30de70ac25b7-kube-api-access-spkxj\") pod \"maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt\" (UID: \"a7f1998f-5d7a-493b-bf98-30de70ac25b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt"
Oct 30 23:57:16.724939 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:16.724907    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt"
Oct 30 23:57:17.110838 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:17.110809    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt"]
Oct 30 23:57:17.114236 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:57:17.114209    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f1998f_5d7a_493b_bf98_30de70ac25b7.slice/crio-4e3f2dd8c1609584c157a22fdfbd1b083a93d7a944885bcec3577d907d4cf7f3 WatchSource:0}: Error finding container 4e3f2dd8c1609584c157a22fdfbd1b083a93d7a944885bcec3577d907d4cf7f3: Status 404 returned error can't find the container with id 4e3f2dd8c1609584c157a22fdfbd1b083a93d7a944885bcec3577d907d4cf7f3
Oct 30 23:57:17.331379 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:17.331353    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt" event={"ID":"a7f1998f-5d7a-493b-bf98-30de70ac25b7","Type":"ContainerStarted","Data":"4e3f2dd8c1609584c157a22fdfbd1b083a93d7a944885bcec3577d907d4cf7f3"}
Oct 30 23:57:17.590159 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:57:17.590126    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:57:18.297776 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:18.297746    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e7542f5-5c45-4cf1-b0b0-0f079709f0d4" path="/var/lib/kubelet/pods/4e7542f5-5c45-4cf1-b0b0-0f079709f0d4/volumes"
Oct 30 23:57:18.338628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:18.338583    5078 generic.go:356] "Generic (PLEG): container finished" podID="a7f1998f-5d7a-493b-bf98-30de70ac25b7" containerID="035ab9801fd6a064269477dcc0a84ed696c91acf32cbe710a754a0ae1dd154d8" exitCode=0
Oct 30 23:57:18.338780 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:18.338673    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt" event={"ID":"a7f1998f-5d7a-493b-bf98-30de70ac25b7","Type":"ContainerDied","Data":"035ab9801fd6a064269477dcc0a84ed696c91acf32cbe710a754a0ae1dd154d8"}
Oct 30 23:57:19.472821 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:19.472799    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt"
Oct 30 23:57:19.589421 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:19.589393    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7f1998f-5d7a-493b-bf98-30de70ac25b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f1998f-5d7a-493b-bf98-30de70ac25b7-maas-api-xks-ca\") pod \"a7f1998f-5d7a-493b-bf98-30de70ac25b7\" (UID: \"a7f1998f-5d7a-493b-bf98-30de70ac25b7\") "
Oct 30 23:57:19.589540 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:19.589431    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f1998f-5d7a-493b-bf98-30de70ac25b7-kube-api-access-spkxj\" (UniqueName: \"kubernetes.io/projected/a7f1998f-5d7a-493b-bf98-30de70ac25b7-kube-api-access-spkxj\") pod \"a7f1998f-5d7a-493b-bf98-30de70ac25b7\" (UID: \"a7f1998f-5d7a-493b-bf98-30de70ac25b7\") "
Oct 30 23:57:19.589540 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:19.589496    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f1998f-5d7a-493b-bf98-30de70ac25b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f1998f-5d7a-493b-bf98-30de70ac25b7-maas-api-openshift-ca\") pod \"a7f1998f-5d7a-493b-bf98-30de70ac25b7\" (UID: \"a7f1998f-5d7a-493b-bf98-30de70ac25b7\") "
Oct 30 23:57:19.590678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:19.590650    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f1998f-5d7a-493b-bf98-30de70ac25b7-maas-api-openshift-ca" pod "a7f1998f-5d7a-493b-bf98-30de70ac25b7" (UID: "a7f1998f-5d7a-493b-bf98-30de70ac25b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:19.591996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:19.591970    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f1998f-5d7a-493b-bf98-30de70ac25b7-kube-api-access-spkxj" pod "a7f1998f-5d7a-493b-bf98-30de70ac25b7" (UID: "a7f1998f-5d7a-493b-bf98-30de70ac25b7"). InnerVolumeSpecName "kube-api-access-spkxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:19.591996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:19.591973    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f1998f-5d7a-493b-bf98-30de70ac25b7-maas-api-xks-ca" pod "a7f1998f-5d7a-493b-bf98-30de70ac25b7" (UID: "a7f1998f-5d7a-493b-bf98-30de70ac25b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:57:19.691159 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:19.691131    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a7f1998f-5d7a-493b-bf98-30de70ac25b7-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:19.691255 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:19.691164    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a7f1998f-5d7a-493b-bf98-30de70ac25b7-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:19.691255 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:19.691174    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spkxj\" (UniqueName: \"kubernetes.io/projected/a7f1998f-5d7a-493b-bf98-30de70ac25b7-kube-api-access-spkxj\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:20.349928 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.349893    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt" event={"ID":"a7f1998f-5d7a-493b-bf98-30de70ac25b7","Type":"ContainerDied","Data":"4e3f2dd8c1609584c157a22fdfbd1b083a93d7a944885bcec3577d907d4cf7f3"}
Oct 30 23:57:20.349928 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.349910    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt"
Oct 30 23:57:20.350143 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.349943    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e3f2dd8c1609584c157a22fdfbd1b083a93d7a944885bcec3577d907d4cf7f3"
Oct 30 23:57:20.651368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.651348    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:57:20.705488 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.705461    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-workload-socket\") pod \"c8b4ee1d-90dd-416a-9955-cc54ad064723\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") "
Oct 30 23:57:20.705625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.705523    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-podinfo\") pod \"c8b4ee1d-90dd-416a-9955-cc54ad064723\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") "
Oct 30 23:57:20.705625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.705573    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-data\") pod \"c8b4ee1d-90dd-416a-9955-cc54ad064723\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") "
Oct 30 23:57:20.705625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.705620    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-token\" (UniqueName: \"kubernetes.io/projected/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-token\") pod \"c8b4ee1d-90dd-416a-9955-cc54ad064723\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") "
Oct 30 23:57:20.705747 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.705717    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-workload-socket" pod "c8b4ee1d-90dd-416a-9955-cc54ad064723" (UID: "c8b4ee1d-90dd-416a-9955-cc54ad064723"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:20.705805 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.705762    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-envoy\") pod \"c8b4ee1d-90dd-416a-9955-cc54ad064723\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") "
Oct 30 23:57:20.705859 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.705821    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-credential-socket\") pod \"c8b4ee1d-90dd-416a-9955-cc54ad064723\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") "
Oct 30 23:57:20.705859 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.705848    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-workload-certs\") pod \"c8b4ee1d-90dd-416a-9955-cc54ad064723\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") "
Oct 30 23:57:20.705979 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.705884    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b4ee1d-90dd-416a-9955-cc54ad064723-kube-api-access-lpdzv\" (UniqueName: \"kubernetes.io/projected/c8b4ee1d-90dd-416a-9955-cc54ad064723-kube-api-access-lpdzv\") pod \"c8b4ee1d-90dd-416a-9955-cc54ad064723\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") "
Oct 30 23:57:20.705979 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.705911    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-data" pod "c8b4ee1d-90dd-416a-9955-cc54ad064723" (UID: "c8b4ee1d-90dd-416a-9955-cc54ad064723"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:20.705979 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.705965    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b4ee1d-90dd-416a-9955-cc54ad064723-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b4ee1d-90dd-416a-9955-cc54ad064723-istiod-ca-cert\") pod \"c8b4ee1d-90dd-416a-9955-cc54ad064723\" (UID: \"c8b4ee1d-90dd-416a-9955-cc54ad064723\") "
Oct 30 23:57:20.706381 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.706294    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:20.706381 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.706316    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:20.706381 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.706352    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-credential-socket" pod "c8b4ee1d-90dd-416a-9955-cc54ad064723" (UID: "c8b4ee1d-90dd-416a-9955-cc54ad064723"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:20.706641 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.706420    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-workload-certs" pod "c8b4ee1d-90dd-416a-9955-cc54ad064723" (UID: "c8b4ee1d-90dd-416a-9955-cc54ad064723"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:20.706641 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.706465    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b4ee1d-90dd-416a-9955-cc54ad064723-istiod-ca-cert" pod "c8b4ee1d-90dd-416a-9955-cc54ad064723" (UID: "c8b4ee1d-90dd-416a-9955-cc54ad064723"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:20.708138 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.708112    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-envoy" pod "c8b4ee1d-90dd-416a-9955-cc54ad064723" (UID: "c8b4ee1d-90dd-416a-9955-cc54ad064723"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:20.708475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.708451    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-podinfo" pod "c8b4ee1d-90dd-416a-9955-cc54ad064723" (UID: "c8b4ee1d-90dd-416a-9955-cc54ad064723"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:57:20.708475 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.708454    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-token" pod "c8b4ee1d-90dd-416a-9955-cc54ad064723" (UID: "c8b4ee1d-90dd-416a-9955-cc54ad064723"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:20.708628 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.708533    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b4ee1d-90dd-416a-9955-cc54ad064723-kube-api-access-lpdzv" pod "c8b4ee1d-90dd-416a-9955-cc54ad064723" (UID: "c8b4ee1d-90dd-416a-9955-cc54ad064723"). InnerVolumeSpecName "kube-api-access-lpdzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:20.807450 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.807427    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:20.807450 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.807448    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:20.807634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.807459    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:20.807634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.807468    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:20.807634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.807476    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8b4ee1d-90dd-416a-9955-cc54ad064723-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:20.807634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.807484    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpdzv\" (UniqueName: \"kubernetes.io/projected/c8b4ee1d-90dd-416a-9955-cc54ad064723-kube-api-access-lpdzv\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:20.807634 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.807493    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8b4ee1d-90dd-416a-9955-cc54ad064723-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:20.859852 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:20.859791    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"]
Oct 30 23:57:21.357211 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:21.357179    5078 generic.go:356] "Generic (PLEG): container finished" podID="c8b4ee1d-90dd-416a-9955-cc54ad064723" containerID="2d792d082bc075db64d00d86bd328798a1a312e75575061d5a9f48639fc4681c" exitCode=0
Oct 30 23:57:21.357367 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:21.357223    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c" event={"ID":"c8b4ee1d-90dd-416a-9955-cc54ad064723","Type":"ContainerDied","Data":"2d792d082bc075db64d00d86bd328798a1a312e75575061d5a9f48639fc4681c"}
Oct 30 23:57:21.357367 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:21.357232    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"
Oct 30 23:57:21.357367 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:21.357265    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c" event={"ID":"c8b4ee1d-90dd-416a-9955-cc54ad064723","Type":"ContainerDied","Data":"d4348bfcc56bbab9da49811ed7495a76be6775175b069934462e48e4cc5dc6a3"}
Oct 30 23:57:21.357367 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:21.357286    5078 scope.go:122] "RemoveContainer" containerID="2d792d082bc075db64d00d86bd328798a1a312e75575061d5a9f48639fc4681c"
Oct 30 23:57:21.367136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:21.367118    5078 scope.go:122] "RemoveContainer" containerID="12b24b5e11c1cd12348c0a391bd1e210ffc895b878b5be170495658ea95ddb13"
Oct 30 23:57:21.375893 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:21.375875    5078 scope.go:122] "RemoveContainer" containerID="2d792d082bc075db64d00d86bd328798a1a312e75575061d5a9f48639fc4681c"
Oct 30 23:57:21.376181 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:57:21.376164    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d792d082bc075db64d00d86bd328798a1a312e75575061d5a9f48639fc4681c\": container with ID starting with 2d792d082bc075db64d00d86bd328798a1a312e75575061d5a9f48639fc4681c not found: ID does not exist" containerID="2d792d082bc075db64d00d86bd328798a1a312e75575061d5a9f48639fc4681c"
Oct 30 23:57:21.376230 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:21.376190    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d792d082bc075db64d00d86bd328798a1a312e75575061d5a9f48639fc4681c"} err="failed to get container status \"2d792d082bc075db64d00d86bd328798a1a312e75575061d5a9f48639fc4681c\": rpc error: code = NotFound desc = could not find container \"2d792d082bc075db64d00d86bd328798a1a312e75575061d5a9f48639fc4681c\": container with ID starting with 2d792d082bc075db64d00d86bd328798a1a312e75575061d5a9f48639fc4681c not found: ID does not exist"
Oct 30 23:57:21.376230 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:21.376213    5078 scope.go:122] "RemoveContainer" containerID="12b24b5e11c1cd12348c0a391bd1e210ffc895b878b5be170495658ea95ddb13"
Oct 30 23:57:21.376464 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:57:21.376446    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12b24b5e11c1cd12348c0a391bd1e210ffc895b878b5be170495658ea95ddb13\": container with ID starting with 12b24b5e11c1cd12348c0a391bd1e210ffc895b878b5be170495658ea95ddb13 not found: ID does not exist" containerID="12b24b5e11c1cd12348c0a391bd1e210ffc895b878b5be170495658ea95ddb13"
Oct 30 23:57:21.376512 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:21.376471    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12b24b5e11c1cd12348c0a391bd1e210ffc895b878b5be170495658ea95ddb13"} err="failed to get container status \"12b24b5e11c1cd12348c0a391bd1e210ffc895b878b5be170495658ea95ddb13\": rpc error: code = NotFound desc = could not find container \"12b24b5e11c1cd12348c0a391bd1e210ffc895b878b5be170495658ea95ddb13\": container with ID starting with 12b24b5e11c1cd12348c0a391bd1e210ffc895b878b5be170495658ea95ddb13 not found: ID does not exist"
Oct 30 23:57:21.522945 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:21.522906    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"]
Oct 30 23:57:21.573530 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:21.573493    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-3be34b-openshift-default-954d88667-nr89c"]
Oct 30 23:57:22.296321 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:22.296287    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b4ee1d-90dd-416a-9955-cc54ad064723" path="/var/lib/kubelet/pods/c8b4ee1d-90dd-416a-9955-cc54ad064723/volumes"
Oct 30 23:57:23.871021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:23.870988    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc"]
Oct 30 23:57:23.977332 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:23.977301    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj"]
Oct 30 23:57:27.698851 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:27.698813    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7"]
Oct 30 23:57:28.872152 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:28.872095    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc" podUID="ca4df19e-725c-4529-8028-18615a736d0b" containerName="payload-processing" containerID="cri-o://1feaad350f7a51b88115a1cf473807e516ff1ea7864fcb218093b67b4b7328bd" gracePeriod=25
Oct 30 23:57:28.978104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:28.978064    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj" podUID="5e4686b5-2668-4680-b215-432e012c3348" containerName="payload-pre-processing" containerID="cri-o://a2d3c35f37b9d2e3fb6607c626cbf188585434e08d8f5bd7d49def7d68587fe5" gracePeriod=25
Oct 30 23:57:30.612272 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:30.612228    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt"]
Oct 30 23:57:30.661427 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:30.661396    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-333b59-57mzt"]
Oct 30 23:57:30.928225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:30.928122    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main" containerID="cri-o://4d143441d651e9e12883eaea28707b319726b018a9b14f476e90f8ee95885d62" gracePeriod=45
Oct 30 23:57:30.943059 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:30.943024    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main" containerID="cri-o://2dbc8a5b2c0a6b18be31937a973466fed6332a57ca992193330c309848b0aacb" gracePeriod=45
Oct 30 23:57:31.193227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.193076    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:57:31.235957 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.235928    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:57:31.310736 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.310705    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bf2b89c-d52c-41bb-a594-4f87590ba512-kube-api-access-p7mcp\" (UniqueName: \"kubernetes.io/projected/5bf2b89c-d52c-41bb-a594-4f87590ba512-kube-api-access-p7mcp\") pod \"5bf2b89c-d52c-41bb-a594-4f87590ba512\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") "
Oct 30 23:57:31.310922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.310752    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-home\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-home\") pod \"5bf2b89c-d52c-41bb-a594-4f87590ba512\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") "
Oct 30 23:57:31.310922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.310771    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-model-cache\") pod \"f9f19c42-fd97-4365-8494-92afa52fa4d3\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") "
Oct 30 23:57:31.310922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.310797    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-dshm\") pod \"5bf2b89c-d52c-41bb-a594-4f87590ba512\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") "
Oct 30 23:57:31.310922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.310824    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9f19c42-fd97-4365-8494-92afa52fa4d3-kube-api-access-s9rsq\" (UniqueName: \"kubernetes.io/projected/f9f19c42-fd97-4365-8494-92afa52fa4d3-kube-api-access-s9rsq\") pod \"f9f19c42-fd97-4365-8494-92afa52fa4d3\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") "
Oct 30 23:57:31.310922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.310861    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9f19c42-fd97-4365-8494-92afa52fa4d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/f9f19c42-fd97-4365-8494-92afa52fa4d3-tls-certs\") pod \"f9f19c42-fd97-4365-8494-92afa52fa4d3\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") "
Oct 30 23:57:31.310922 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.310914    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bf2b89c-d52c-41bb-a594-4f87590ba512-tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf2b89c-d52c-41bb-a594-4f87590ba512-tls-certs\") pod \"5bf2b89c-d52c-41bb-a594-4f87590ba512\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") "
Oct 30 23:57:31.311276 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.311000    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-dshm\") pod \"f9f19c42-fd97-4365-8494-92afa52fa4d3\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") "
Oct 30 23:57:31.311276 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.311035    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-tmp-dir\") pod \"5bf2b89c-d52c-41bb-a594-4f87590ba512\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") "
Oct 30 23:57:31.311276 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.311044    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-model-cache" pod "f9f19c42-fd97-4365-8494-92afa52fa4d3" (UID: "f9f19c42-fd97-4365-8494-92afa52fa4d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:31.311276 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.311054    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-home" pod "5bf2b89c-d52c-41bb-a594-4f87590ba512" (UID: "5bf2b89c-d52c-41bb-a594-4f87590ba512"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:31.311276 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.311066    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-model-cache\") pod \"5bf2b89c-d52c-41bb-a594-4f87590ba512\" (UID: \"5bf2b89c-d52c-41bb-a594-4f87590ba512\") "
Oct 30 23:57:31.311276 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.311101    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-home\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-home\") pod \"f9f19c42-fd97-4365-8494-92afa52fa4d3\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") "
Oct 30 23:57:31.311276 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.311177    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-tmp-dir\") pod \"f9f19c42-fd97-4365-8494-92afa52fa4d3\" (UID: \"f9f19c42-fd97-4365-8494-92afa52fa4d3\") "
Oct 30 23:57:31.311738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.311469    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-model-cache" pod "5bf2b89c-d52c-41bb-a594-4f87590ba512" (UID: "5bf2b89c-d52c-41bb-a594-4f87590ba512"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:31.311738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.311534    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:31.311738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.311557    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:31.311738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.311681    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-tmp-dir" pod "5bf2b89c-d52c-41bb-a594-4f87590ba512" (UID: "5bf2b89c-d52c-41bb-a594-4f87590ba512"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:31.311964 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.311871    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-tmp-dir" pod "f9f19c42-fd97-4365-8494-92afa52fa4d3" (UID: "f9f19c42-fd97-4365-8494-92afa52fa4d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:31.312023 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.311966    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-home" pod "f9f19c42-fd97-4365-8494-92afa52fa4d3" (UID: "f9f19c42-fd97-4365-8494-92afa52fa4d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:31.313532 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.313502    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-dshm" pod "f9f19c42-fd97-4365-8494-92afa52fa4d3" (UID: "f9f19c42-fd97-4365-8494-92afa52fa4d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:31.313695 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.313673    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-dshm" pod "5bf2b89c-d52c-41bb-a594-4f87590ba512" (UID: "5bf2b89c-d52c-41bb-a594-4f87590ba512"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:31.313695 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.313682    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf2b89c-d52c-41bb-a594-4f87590ba512-tls-certs" pod "5bf2b89c-d52c-41bb-a594-4f87590ba512" (UID: "5bf2b89c-d52c-41bb-a594-4f87590ba512"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:57:31.313826 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.313762    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9f19c42-fd97-4365-8494-92afa52fa4d3-kube-api-access-s9rsq" pod "f9f19c42-fd97-4365-8494-92afa52fa4d3" (UID: "f9f19c42-fd97-4365-8494-92afa52fa4d3"). InnerVolumeSpecName "kube-api-access-s9rsq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:31.313826 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.313781    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf2b89c-d52c-41bb-a594-4f87590ba512-kube-api-access-p7mcp" pod "5bf2b89c-d52c-41bb-a594-4f87590ba512" (UID: "5bf2b89c-d52c-41bb-a594-4f87590ba512"). InnerVolumeSpecName "kube-api-access-p7mcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:31.314130 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.314110    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9f19c42-fd97-4365-8494-92afa52fa4d3-tls-certs" pod "f9f19c42-fd97-4365-8494-92afa52fa4d3" (UID: "f9f19c42-fd97-4365-8494-92afa52fa4d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:57:31.412729 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.412699    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:31.412729 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.412728    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9rsq\" (UniqueName: \"kubernetes.io/projected/f9f19c42-fd97-4365-8494-92afa52fa4d3-kube-api-access-s9rsq\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:31.412946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.412738    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9f19c42-fd97-4365-8494-92afa52fa4d3-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:31.412946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.412748    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf2b89c-d52c-41bb-a594-4f87590ba512-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:31.412946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.412756    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:31.412946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.412764    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:31.412946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.412772    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf2b89c-d52c-41bb-a594-4f87590ba512-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:31.412946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.412781    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:31.412946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.412789    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9f19c42-fd97-4365-8494-92afa52fa4d3-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:31.412946 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.412797    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7mcp\" (UniqueName: \"kubernetes.io/projected/5bf2b89c-d52c-41bb-a594-4f87590ba512-kube-api-access-p7mcp\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:31.428825 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.428793    5078 generic.go:356] "Generic (PLEG): container finished" podID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerID="4d143441d651e9e12883eaea28707b319726b018a9b14f476e90f8ee95885d62" exitCode=0
Oct 30 23:57:31.428999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.428896    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"
Oct 30 23:57:31.428999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.428920    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" event={"ID":"5bf2b89c-d52c-41bb-a594-4f87590ba512","Type":"ContainerDied","Data":"4d143441d651e9e12883eaea28707b319726b018a9b14f476e90f8ee95885d62"}
Oct 30 23:57:31.428999 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.428984    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs" event={"ID":"5bf2b89c-d52c-41bb-a594-4f87590ba512","Type":"ContainerDied","Data":"c81dd4b3d6a721c70e2359ab625133ffca22e95ed734e492fcae866ea21bdcf1"}
Oct 30 23:57:31.429144 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.429008    5078 scope.go:122] "RemoveContainer" containerID="4d143441d651e9e12883eaea28707b319726b018a9b14f476e90f8ee95885d62"
Oct 30 23:57:31.430524 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.430497    5078 generic.go:356] "Generic (PLEG): container finished" podID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerID="2dbc8a5b2c0a6b18be31937a973466fed6332a57ca992193330c309848b0aacb" exitCode=0
Oct 30 23:57:31.430648 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.430553    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"
Oct 30 23:57:31.430648 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.430585    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" event={"ID":"f9f19c42-fd97-4365-8494-92afa52fa4d3","Type":"ContainerDied","Data":"2dbc8a5b2c0a6b18be31937a973466fed6332a57ca992193330c309848b0aacb"}
Oct 30 23:57:31.430753 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.430667    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6" event={"ID":"f9f19c42-fd97-4365-8494-92afa52fa4d3","Type":"ContainerDied","Data":"9f0752f099b50a993a65c6e6d6b08b3dc38326942f0c50e9034b37156eaee820"}
Oct 30 23:57:31.439858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.439841    5078 scope.go:122] "RemoveContainer" containerID="4d143441d651e9e12883eaea28707b319726b018a9b14f476e90f8ee95885d62"
Oct 30 23:57:31.440196 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:57:31.440177    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d143441d651e9e12883eaea28707b319726b018a9b14f476e90f8ee95885d62\": container with ID starting with 4d143441d651e9e12883eaea28707b319726b018a9b14f476e90f8ee95885d62 not found: ID does not exist" containerID="4d143441d651e9e12883eaea28707b319726b018a9b14f476e90f8ee95885d62"
Oct 30 23:57:31.440301 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.440213    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d143441d651e9e12883eaea28707b319726b018a9b14f476e90f8ee95885d62"} err="failed to get container status \"4d143441d651e9e12883eaea28707b319726b018a9b14f476e90f8ee95885d62\": rpc error: code = NotFound desc = could not find container \"4d143441d651e9e12883eaea28707b319726b018a9b14f476e90f8ee95885d62\": container with ID starting with 4d143441d651e9e12883eaea28707b319726b018a9b14f476e90f8ee95885d62 not found: ID does not exist"
Oct 30 23:57:31.440301 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.440240    5078 scope.go:122] "RemoveContainer" containerID="2dbc8a5b2c0a6b18be31937a973466fed6332a57ca992193330c309848b0aacb"
Oct 30 23:57:31.451295 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.451271    5078 scope.go:122] "RemoveContainer" containerID="2dbc8a5b2c0a6b18be31937a973466fed6332a57ca992193330c309848b0aacb"
Oct 30 23:57:31.451646 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:57:31.451579    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dbc8a5b2c0a6b18be31937a973466fed6332a57ca992193330c309848b0aacb\": container with ID starting with 2dbc8a5b2c0a6b18be31937a973466fed6332a57ca992193330c309848b0aacb not found: ID does not exist" containerID="2dbc8a5b2c0a6b18be31937a973466fed6332a57ca992193330c309848b0aacb"
Oct 30 23:57:31.451759 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.451644    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dbc8a5b2c0a6b18be31937a973466fed6332a57ca992193330c309848b0aacb"} err="failed to get container status \"2dbc8a5b2c0a6b18be31937a973466fed6332a57ca992193330c309848b0aacb\": rpc error: code = NotFound desc = could not find container \"2dbc8a5b2c0a6b18be31937a973466fed6332a57ca992193330c309848b0aacb\": container with ID starting with 2dbc8a5b2c0a6b18be31937a973466fed6332a57ca992193330c309848b0aacb not found: ID does not exist"
Oct 30 23:57:31.554694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.554657    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"]
Oct 30 23:57:31.560032 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.560007    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/simulator-w5-5dc3c0-kserve-5965574944-vtsp6"]
Oct 30 23:57:31.672265 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.672234    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"]
Oct 30 23:57:31.675944 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:31.675922    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-5dc3c0/distinct-w5-5dc3c0-kserve-8d97b6bcb-8fmfs"]
Oct 30 23:57:32.296797 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:32.296766    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f1ae927-6114-44e3-b7c1-2f1ceb19932c" path="/var/lib/kubelet/pods/4f1ae927-6114-44e3-b7c1-2f1ceb19932c/volumes"
Oct 30 23:57:32.297178 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:32.297165    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" path="/var/lib/kubelet/pods/5bf2b89c-d52c-41bb-a594-4f87590ba512/volumes"
Oct 30 23:57:32.297511 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:32.297499    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" path="/var/lib/kubelet/pods/f9f19c42-fd97-4365-8494-92afa52fa4d3/volumes"
Oct 30 23:57:32.699383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:32.699293    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7" podUID="2a6cb384-8d9a-4b7d-a936-b8c556d30170" containerName="payload-processing" containerID="cri-o://6978d4e0aafebfc7c1913d57309fe7e93fda6e19b2e996c3829b41d82bff56a8" gracePeriod=25
Oct 30 23:57:33.851610 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:33.851558    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj" podUID="5e4686b5-2668-4680-b215-432e012c3348" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:57:33.851610 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:33.851585    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc" podUID="ca4df19e-725c-4529-8028-18615a736d0b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:57:34.980175 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:34.980142    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r"]
Oct 30 23:57:34.980568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:34.980517    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c8b4ee1d-90dd-416a-9955-cc54ad064723" containerName="istio-proxy"
Oct 30 23:57:34.980568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:34.980529    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b4ee1d-90dd-416a-9955-cc54ad064723" containerName="istio-proxy"
Oct 30 23:57:34.980568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:34.980540    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main"
Oct 30 23:57:34.980568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:34.980545    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main"
Oct 30 23:57:34.980568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:34.980552    5078 cpu_manager.go:414] "containerMap: removing container" podUID="c8b4ee1d-90dd-416a-9955-cc54ad064723" containerName="istio-proxy"
Oct 30 23:57:34.980568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:34.980559    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b4ee1d-90dd-416a-9955-cc54ad064723" containerName="istio-proxy"
Oct 30 23:57:34.980800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:34.980583    5078 cpu_manager.go:414] "containerMap: removing container" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main"
Oct 30 23:57:34.980800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:34.980588    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main"
Oct 30 23:57:34.980800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:34.980625    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a7f1998f-5d7a-493b-bf98-30de70ac25b7" containerName="revoke-keys"
Oct 30 23:57:34.980800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:34.980631    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f1998f-5d7a-493b-bf98-30de70ac25b7" containerName="revoke-keys"
Oct 30 23:57:34.980800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:34.980683    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bf2b89c-d52c-41bb-a594-4f87590ba512" containerName="main"
Oct 30 23:57:34.980800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:34.980697    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a61efb0a-769e-4659-9452-3302535611c1" containerName="istio-proxy"
Oct 30 23:57:34.980800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:34.980705    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f1998f-5d7a-493b-bf98-30de70ac25b7" containerName="revoke-keys"
Oct 30 23:57:34.980800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:34.980711    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b4ee1d-90dd-416a-9955-cc54ad064723" containerName="istio-proxy"
Oct 30 23:57:34.980800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:34.980716    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9f19c42-fd97-4365-8494-92afa52fa4d3" containerName="main"
Oct 30 23:57:34.980800 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:34.980722    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b4ee1d-90dd-416a-9955-cc54ad064723" containerName="istio-proxy"
Oct 30 23:57:34.985399 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:34.985380    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r"
Oct 30 23:57:35.044279 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.044253    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:57:35.044422 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.044379    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:57:35.094221 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.094192    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:57:35.094460 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.094445    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:57:35.096171 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.096152    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r"]
Oct 30 23:57:35.149749 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.149708    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcp6r\" (UniqueName: \"kubernetes.io/projected/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-kube-api-access-dcp6r\") pod \"maas-api-revoke-sub-0b150622-sbc6r\" (UID: \"ec9d021b-6fa0-4df4-8021-14ab08a20c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r"
Oct 30 23:57:35.149919 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.149759    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b150622-sbc6r\" (UID: \"ec9d021b-6fa0-4df4-8021-14ab08a20c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r"
Oct 30 23:57:35.149919 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.149873    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b150622-sbc6r\" (UID: \"ec9d021b-6fa0-4df4-8021-14ab08a20c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r"
Oct 30 23:57:35.251372 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.251279    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcp6r\" (UniqueName: \"kubernetes.io/projected/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-kube-api-access-dcp6r\") pod \"maas-api-revoke-sub-0b150622-sbc6r\" (UID: \"ec9d021b-6fa0-4df4-8021-14ab08a20c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r"
Oct 30 23:57:35.251372 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.251357    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b150622-sbc6r\" (UID: \"ec9d021b-6fa0-4df4-8021-14ab08a20c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r"
Oct 30 23:57:35.251644 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.251441    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b150622-sbc6r\" (UID: \"ec9d021b-6fa0-4df4-8021-14ab08a20c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r"
Oct 30 23:57:35.252133 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.252104    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b150622-sbc6r\" (UID: \"ec9d021b-6fa0-4df4-8021-14ab08a20c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r"
Oct 30 23:57:35.253405 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.253377    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r"]
Oct 30 23:57:35.253878 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.253859    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b150622-sbc6r\" (UID: \"ec9d021b-6fa0-4df4-8021-14ab08a20c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r"
Oct 30 23:57:35.256923 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.256903    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r"
Oct 30 23:57:35.309474 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.309452    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcp6r\" (UniqueName: \"kubernetes.io/projected/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-kube-api-access-dcp6r\") pod \"maas-api-revoke-sub-0b150622-sbc6r\" (UID: \"ec9d021b-6fa0-4df4-8021-14ab08a20c3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r"
Oct 30 23:57:35.316921 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.316898    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r"]
Oct 30 23:57:35.352225 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.352191    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f189cd0f-f06e-4495-931c-38aac8abb98a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaea140b-2nf2r\" (UID: \"f189cd0f-f06e-4495-931c-38aac8abb98a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r"
Oct 30 23:57:35.352381 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.352235    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccp72\" (UniqueName: \"kubernetes.io/projected/f189cd0f-f06e-4495-931c-38aac8abb98a-kube-api-access-ccp72\") pod \"maas-api-revoke-sub-eaea140b-2nf2r\" (UID: \"f189cd0f-f06e-4495-931c-38aac8abb98a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r"
Oct 30 23:57:35.352381 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.352304    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f189cd0f-f06e-4495-931c-38aac8abb98a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaea140b-2nf2r\" (UID: \"f189cd0f-f06e-4495-931c-38aac8abb98a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r"
Oct 30 23:57:35.453038 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.453009    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f189cd0f-f06e-4495-931c-38aac8abb98a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaea140b-2nf2r\" (UID: \"f189cd0f-f06e-4495-931c-38aac8abb98a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r"
Oct 30 23:57:35.453266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.453048    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccp72\" (UniqueName: \"kubernetes.io/projected/f189cd0f-f06e-4495-931c-38aac8abb98a-kube-api-access-ccp72\") pod \"maas-api-revoke-sub-eaea140b-2nf2r\" (UID: \"f189cd0f-f06e-4495-931c-38aac8abb98a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r"
Oct 30 23:57:35.453266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.453092    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f189cd0f-f06e-4495-931c-38aac8abb98a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaea140b-2nf2r\" (UID: \"f189cd0f-f06e-4495-931c-38aac8abb98a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r"
Oct 30 23:57:35.453750 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.453727    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f189cd0f-f06e-4495-931c-38aac8abb98a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eaea140b-2nf2r\" (UID: \"f189cd0f-f06e-4495-931c-38aac8abb98a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r"
Oct 30 23:57:35.455499 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.455480    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f189cd0f-f06e-4495-931c-38aac8abb98a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eaea140b-2nf2r\" (UID: \"f189cd0f-f06e-4495-931c-38aac8abb98a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r"
Oct 30 23:57:35.511348 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.511279    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccp72\" (UniqueName: \"kubernetes.io/projected/f189cd0f-f06e-4495-931c-38aac8abb98a-kube-api-access-ccp72\") pod \"maas-api-revoke-sub-eaea140b-2nf2r\" (UID: \"f189cd0f-f06e-4495-931c-38aac8abb98a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r"
Oct 30 23:57:35.572649 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.572618    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r"
Oct 30 23:57:35.597929 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.597897    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r"
Oct 30 23:57:35.879485 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.879439    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main" containerID="cri-o://b3df52657d928af74af45a7d355b3d186621068ddf0f0caa321bf87a8c704817" gracePeriod=45
Oct 30 23:57:35.948227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.948197    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r"]
Oct 30 23:57:35.988218 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:57:35.988190    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf189cd0f_f06e_4495_931c_38aac8abb98a.slice/crio-bb3d2402b2aed2dc549fff39215518bdb953d49704ffa186e60c9a83fe549b28 WatchSource:0}: Error finding container bb3d2402b2aed2dc549fff39215518bdb953d49704ffa186e60c9a83fe549b28: Status 404 returned error can't find the container with id bb3d2402b2aed2dc549fff39215518bdb953d49704ffa186e60c9a83fe549b28
Oct 30 23:57:35.999104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:35.998993    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r"]
Oct 30 23:57:36.000552 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:57:36.000474    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec9d021b_6fa0_4df4_8021_14ab08a20c3e.slice/crio-f2b3e30f53b7ff764b9839b3f09a40f75062aab297601c9b54ae65646d71e110 WatchSource:0}: Error finding container f2b3e30f53b7ff764b9839b3f09a40f75062aab297601c9b54ae65646d71e110: Status 404 returned error can't find the container with id f2b3e30f53b7ff764b9839b3f09a40f75062aab297601c9b54ae65646d71e110
Oct 30 23:57:36.121131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.121109    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:57:36.260583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.260367    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-model-cache\") pod \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") "
Oct 30 23:57:36.260583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.260439    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-home\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-home\") pod \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") "
Oct 30 23:57:36.260583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.260505    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-dshm\") pod \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") "
Oct 30 23:57:36.260583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.260531    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a87c532-4c80-46b8-98ee-4b07dadba3b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a87c532-4c80-46b8-98ee-4b07dadba3b2-tls-certs\") pod \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") "
Oct 30 23:57:36.260583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.260555    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a87c532-4c80-46b8-98ee-4b07dadba3b2-kube-api-access-k5scf\" (UniqueName: \"kubernetes.io/projected/4a87c532-4c80-46b8-98ee-4b07dadba3b2-kube-api-access-k5scf\") pod \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") "
Oct 30 23:57:36.260583 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.260575    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-tmp-dir\") pod \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\" (UID: \"4a87c532-4c80-46b8-98ee-4b07dadba3b2\") "
Oct 30 23:57:36.261263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.260857    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-model-cache" pod "4a87c532-4c80-46b8-98ee-4b07dadba3b2" (UID: "4a87c532-4c80-46b8-98ee-4b07dadba3b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:36.261263 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.261028    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-tmp-dir" pod "4a87c532-4c80-46b8-98ee-4b07dadba3b2" (UID: "4a87c532-4c80-46b8-98ee-4b07dadba3b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:36.261384 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.261311    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-home" pod "4a87c532-4c80-46b8-98ee-4b07dadba3b2" (UID: "4a87c532-4c80-46b8-98ee-4b07dadba3b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:36.263808 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.263782    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-dshm" pod "4a87c532-4c80-46b8-98ee-4b07dadba3b2" (UID: "4a87c532-4c80-46b8-98ee-4b07dadba3b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:36.264103 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.264079    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a87c532-4c80-46b8-98ee-4b07dadba3b2-tls-certs" pod "4a87c532-4c80-46b8-98ee-4b07dadba3b2" (UID: "4a87c532-4c80-46b8-98ee-4b07dadba3b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:57:36.264227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.264099    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a87c532-4c80-46b8-98ee-4b07dadba3b2-kube-api-access-k5scf" pod "4a87c532-4c80-46b8-98ee-4b07dadba3b2" (UID: "4a87c532-4c80-46b8-98ee-4b07dadba3b2"). InnerVolumeSpecName "kube-api-access-k5scf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:36.361976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.361944    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:36.361976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.361976    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a87c532-4c80-46b8-98ee-4b07dadba3b2-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:36.362201 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.362108    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5scf\" (UniqueName: \"kubernetes.io/projected/4a87c532-4c80-46b8-98ee-4b07dadba3b2-kube-api-access-k5scf\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:36.362201 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.362129    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:36.362201 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.362139    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:36.362201 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.362148    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a87c532-4c80-46b8-98ee-4b07dadba3b2-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:36.462648 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.462455    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r" event={"ID":"f189cd0f-f06e-4495-931c-38aac8abb98a","Type":"ContainerStarted","Data":"cce43484a5d918bcd9064fdef73cc72d4824180579f36b3f25ba6984de35be1e"}
Oct 30 23:57:36.462648 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.462519    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r" event={"ID":"f189cd0f-f06e-4495-931c-38aac8abb98a","Type":"ContainerStarted","Data":"bb3d2402b2aed2dc549fff39215518bdb953d49704ffa186e60c9a83fe549b28"}
Oct 30 23:57:36.464020 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.463993    5078 generic.go:356] "Generic (PLEG): container finished" podID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerID="b3df52657d928af74af45a7d355b3d186621068ddf0f0caa321bf87a8c704817" exitCode=0
Oct 30 23:57:36.464147 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.464037    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"
Oct 30 23:57:36.464147 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.464081    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" event={"ID":"4a87c532-4c80-46b8-98ee-4b07dadba3b2","Type":"ContainerDied","Data":"b3df52657d928af74af45a7d355b3d186621068ddf0f0caa321bf87a8c704817"}
Oct 30 23:57:36.464147 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.464117    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8" event={"ID":"4a87c532-4c80-46b8-98ee-4b07dadba3b2","Type":"ContainerDied","Data":"7004d6a3c0d878959d4449389ea4f6fe8dc88f580c964c5eab4a40d904f38d52"}
Oct 30 23:57:36.464147 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.464142    5078 scope.go:122] "RemoveContainer" containerID="b3df52657d928af74af45a7d355b3d186621068ddf0f0caa321bf87a8c704817"
Oct 30 23:57:36.466110 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.466081    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r" event={"ID":"ec9d021b-6fa0-4df4-8021-14ab08a20c3e","Type":"ContainerStarted","Data":"d2398f29837f450735e03a1d063915728317efe264a6178af705350fe5e9bd55"}
Oct 30 23:57:36.466222 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.466135    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r" event={"ID":"ec9d021b-6fa0-4df4-8021-14ab08a20c3e","Type":"ContainerStarted","Data":"f2b3e30f53b7ff764b9839b3f09a40f75062aab297601c9b54ae65646d71e110"}
Oct 30 23:57:36.474784 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.474764    5078 scope.go:122] "RemoveContainer" containerID="b3df52657d928af74af45a7d355b3d186621068ddf0f0caa321bf87a8c704817"
Oct 30 23:57:36.475102 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:57:36.475084    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3df52657d928af74af45a7d355b3d186621068ddf0f0caa321bf87a8c704817\": container with ID starting with b3df52657d928af74af45a7d355b3d186621068ddf0f0caa321bf87a8c704817 not found: ID does not exist" containerID="b3df52657d928af74af45a7d355b3d186621068ddf0f0caa321bf87a8c704817"
Oct 30 23:57:36.475167 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.475114    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3df52657d928af74af45a7d355b3d186621068ddf0f0caa321bf87a8c704817"} err="failed to get container status \"b3df52657d928af74af45a7d355b3d186621068ddf0f0caa321bf87a8c704817\": rpc error: code = NotFound desc = could not find container \"b3df52657d928af74af45a7d355b3d186621068ddf0f0caa321bf87a8c704817\": container with ID starting with b3df52657d928af74af45a7d355b3d186621068ddf0f0caa321bf87a8c704817 not found: ID does not exist"
Oct 30 23:57:36.581690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.581640    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r" podStartSLOduration=1.5816245740000001 podStartE2EDuration="1.581624574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:36.579184991 +0000 UTC m=+3042.467091344" watchObservedRunningTime="2026-10-30 23:57:36.581624574 +0000 UTC m=+3042.469530931"
Oct 30 23:57:36.694713 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.694676    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"]
Oct 30 23:57:36.699315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.699290    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-aadaa6/distinct-w6-aadaa6-kserve-65f8dcdd5-9bjd8"]
Oct 30 23:57:36.813790 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:36.813739    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r" podStartSLOduration=2.813722984 podStartE2EDuration="2.813722984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:36.812979211 +0000 UTC m=+3042.700885564" watchObservedRunningTime="2026-10-30 23:57:36.813722984 +0000 UTC m=+3042.701629417"
Oct 30 23:57:37.474424 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:37.474390    5078 generic.go:356] "Generic (PLEG): container finished" podID="f189cd0f-f06e-4495-931c-38aac8abb98a" containerID="cce43484a5d918bcd9064fdef73cc72d4824180579f36b3f25ba6984de35be1e" exitCode=0
Oct 30 23:57:37.474888 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:37.474470    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r" event={"ID":"f189cd0f-f06e-4495-931c-38aac8abb98a","Type":"ContainerDied","Data":"cce43484a5d918bcd9064fdef73cc72d4824180579f36b3f25ba6984de35be1e"}
Oct 30 23:57:37.476913 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:37.476892    5078 generic.go:356] "Generic (PLEG): container finished" podID="ec9d021b-6fa0-4df4-8021-14ab08a20c3e" containerID="d2398f29837f450735e03a1d063915728317efe264a6178af705350fe5e9bd55" exitCode=0
Oct 30 23:57:37.477027 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:37.476921    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r" event={"ID":"ec9d021b-6fa0-4df4-8021-14ab08a20c3e","Type":"ContainerDied","Data":"d2398f29837f450735e03a1d063915728317efe264a6178af705350fe5e9bd55"}
Oct 30 23:57:38.296615 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.296569    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" path="/var/lib/kubelet/pods/4a87c532-4c80-46b8-98ee-4b07dadba3b2/volumes"
Oct 30 23:57:38.649198 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.649176    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r"
Oct 30 23:57:38.651411 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.651392    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r"
Oct 30 23:57:38.658969 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.658946    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"]
Oct 30 23:57:38.659144 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.659123    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q" podUID="357e99ba-4145-479f-aa06-cf148c2c52b4" containerName="istio-proxy" containerID="cri-o://a5601e17240d9a3e10c527feece9248f4ed46ed31f13887b782274298471a93d" gracePeriod=30
Oct 30 23:57:38.784809 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.784775    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f189cd0f-f06e-4495-931c-38aac8abb98a-kube-api-access-ccp72\" (UniqueName: \"kubernetes.io/projected/f189cd0f-f06e-4495-931c-38aac8abb98a-kube-api-access-ccp72\") pod \"f189cd0f-f06e-4495-931c-38aac8abb98a\" (UID: \"f189cd0f-f06e-4495-931c-38aac8abb98a\") "
Oct 30 23:57:38.785004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.784865    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f189cd0f-f06e-4495-931c-38aac8abb98a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f189cd0f-f06e-4495-931c-38aac8abb98a-maas-api-openshift-ca\") pod \"f189cd0f-f06e-4495-931c-38aac8abb98a\" (UID: \"f189cd0f-f06e-4495-931c-38aac8abb98a\") "
Oct 30 23:57:38.785004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.784961    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-maas-api-xks-ca\") pod \"ec9d021b-6fa0-4df4-8021-14ab08a20c3e\" (UID: \"ec9d021b-6fa0-4df4-8021-14ab08a20c3e\") "
Oct 30 23:57:38.785004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.784996    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-maas-api-openshift-ca\") pod \"ec9d021b-6fa0-4df4-8021-14ab08a20c3e\" (UID: \"ec9d021b-6fa0-4df4-8021-14ab08a20c3e\") "
Oct 30 23:57:38.785188 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.785039    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f189cd0f-f06e-4495-931c-38aac8abb98a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f189cd0f-f06e-4495-931c-38aac8abb98a-maas-api-xks-ca\") pod \"f189cd0f-f06e-4495-931c-38aac8abb98a\" (UID: \"f189cd0f-f06e-4495-931c-38aac8abb98a\") "
Oct 30 23:57:38.785188 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.785084    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-kube-api-access-dcp6r\" (UniqueName: \"kubernetes.io/projected/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-kube-api-access-dcp6r\") pod \"ec9d021b-6fa0-4df4-8021-14ab08a20c3e\" (UID: \"ec9d021b-6fa0-4df4-8021-14ab08a20c3e\") "
Oct 30 23:57:38.785301 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.785257    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f189cd0f-f06e-4495-931c-38aac8abb98a-maas-api-openshift-ca" pod "f189cd0f-f06e-4495-931c-38aac8abb98a" (UID: "f189cd0f-f06e-4495-931c-38aac8abb98a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:38.785406 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.785361    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-maas-api-openshift-ca" pod "ec9d021b-6fa0-4df4-8021-14ab08a20c3e" (UID: "ec9d021b-6fa0-4df4-8021-14ab08a20c3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:38.785563 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.785457    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f189cd0f-f06e-4495-931c-38aac8abb98a-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:38.785563 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.785484    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:38.787306 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.787277    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f189cd0f-f06e-4495-931c-38aac8abb98a-maas-api-xks-ca" pod "f189cd0f-f06e-4495-931c-38aac8abb98a" (UID: "f189cd0f-f06e-4495-931c-38aac8abb98a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:57:38.787411 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.787371    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-kube-api-access-dcp6r" pod "ec9d021b-6fa0-4df4-8021-14ab08a20c3e" (UID: "ec9d021b-6fa0-4df4-8021-14ab08a20c3e"). InnerVolumeSpecName "kube-api-access-dcp6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:38.787580 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.787562    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f189cd0f-f06e-4495-931c-38aac8abb98a-kube-api-access-ccp72" pod "f189cd0f-f06e-4495-931c-38aac8abb98a" (UID: "f189cd0f-f06e-4495-931c-38aac8abb98a"). InnerVolumeSpecName "kube-api-access-ccp72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:38.787879 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.787859    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-maas-api-xks-ca" pod "ec9d021b-6fa0-4df4-8021-14ab08a20c3e" (UID: "ec9d021b-6fa0-4df4-8021-14ab08a20c3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:57:38.878141 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.878074    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7" podUID="2a6cb384-8d9a-4b7d-a936-b8c556d30170" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:57:38.886407 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.886383    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:38.886520 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.886410    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f189cd0f-f06e-4495-931c-38aac8abb98a-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:38.886520 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.886426    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcp6r\" (UniqueName: \"kubernetes.io/projected/ec9d021b-6fa0-4df4-8021-14ab08a20c3e-kube-api-access-dcp6r\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:38.886520 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:38.886439    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccp72\" (UniqueName: \"kubernetes.io/projected/f189cd0f-f06e-4495-931c-38aac8abb98a-kube-api-access-ccp72\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:39.487902 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.487870    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r"
Oct 30 23:57:39.488101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.487905    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r" event={"ID":"ec9d021b-6fa0-4df4-8021-14ab08a20c3e","Type":"ContainerDied","Data":"f2b3e30f53b7ff764b9839b3f09a40f75062aab297601c9b54ae65646d71e110"}
Oct 30 23:57:39.488101 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.487960    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2b3e30f53b7ff764b9839b3f09a40f75062aab297601c9b54ae65646d71e110"
Oct 30 23:57:39.489255 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.489230    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r"
Oct 30 23:57:39.489420 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.489256    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r" event={"ID":"f189cd0f-f06e-4495-931c-38aac8abb98a","Type":"ContainerDied","Data":"bb3d2402b2aed2dc549fff39215518bdb953d49704ffa186e60c9a83fe549b28"}
Oct 30 23:57:39.489420 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.489297    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb3d2402b2aed2dc549fff39215518bdb953d49704ffa186e60c9a83fe549b28"
Oct 30 23:57:39.697372 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.697337    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g"]
Oct 30 23:57:39.697761 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.697718    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f189cd0f-f06e-4495-931c-38aac8abb98a" containerName="revoke-keys"
Oct 30 23:57:39.697761 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.697731    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f189cd0f-f06e-4495-931c-38aac8abb98a" containerName="revoke-keys"
Oct 30 23:57:39.697761 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.697747    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ec9d021b-6fa0-4df4-8021-14ab08a20c3e" containerName="revoke-keys"
Oct 30 23:57:39.697761 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.697752    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec9d021b-6fa0-4df4-8021-14ab08a20c3e" containerName="revoke-keys"
Oct 30 23:57:39.697761 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.697761    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main"
Oct 30 23:57:39.697949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.697770    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main"
Oct 30 23:57:39.697949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.697828    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f189cd0f-f06e-4495-931c-38aac8abb98a" containerName="revoke-keys"
Oct 30 23:57:39.697949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.697842    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec9d021b-6fa0-4df4-8021-14ab08a20c3e" containerName="revoke-keys"
Oct 30 23:57:39.697949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.697851    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a87c532-4c80-46b8-98ee-4b07dadba3b2" containerName="main"
Oct 30 23:57:39.701253 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.701237    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g"
Oct 30 23:57:39.756735 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.756117    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:57:39.756735 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.756559    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:57:39.756735 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.756625    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:57:39.757036 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.756865    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:57:39.758991 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.758964    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g"]
Oct 30 23:57:39.795074 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.795045    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g\" (UID: \"860e198e-cbb1-4cfd-b7cf-f7838bd5bb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g"
Oct 30 23:57:39.795217 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.795081    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tbng\" (UniqueName: \"kubernetes.io/projected/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-kube-api-access-8tbng\") pod \"maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g\" (UID: \"860e198e-cbb1-4cfd-b7cf-f7838bd5bb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g"
Oct 30 23:57:39.795217 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.795100    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g\" (UID: \"860e198e-cbb1-4cfd-b7cf-f7838bd5bb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g"
Oct 30 23:57:39.896669 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.896640    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g\" (UID: \"860e198e-cbb1-4cfd-b7cf-f7838bd5bb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g"
Oct 30 23:57:39.896839 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.896676    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tbng\" (UniqueName: \"kubernetes.io/projected/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-kube-api-access-8tbng\") pod \"maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g\" (UID: \"860e198e-cbb1-4cfd-b7cf-f7838bd5bb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g"
Oct 30 23:57:39.896839 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.896712    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g\" (UID: \"860e198e-cbb1-4cfd-b7cf-f7838bd5bb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g"
Oct 30 23:57:39.897391 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.897367    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g\" (UID: \"860e198e-cbb1-4cfd-b7cf-f7838bd5bb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g"
Oct 30 23:57:39.899136 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.899119    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g\" (UID: \"860e198e-cbb1-4cfd-b7cf-f7838bd5bb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g"
Oct 30 23:57:39.958893 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:39.958854    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tbng\" (UniqueName: \"kubernetes.io/projected/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-kube-api-access-8tbng\") pod \"maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g\" (UID: \"860e198e-cbb1-4cfd-b7cf-f7838bd5bb33\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g"
Oct 30 23:57:40.013657 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:40.013557    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g"
Oct 30 23:57:40.419872 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:40.419788    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g"]
Oct 30 23:57:40.422085 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:57:40.422056    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860e198e_cbb1_4cfd_b7cf_f7838bd5bb33.slice/crio-8ebef5b44dde7f1077abb3d2ad6020f3120127f7c5d69b49020f46a3fc965cf1 WatchSource:0}: Error finding container 8ebef5b44dde7f1077abb3d2ad6020f3120127f7c5d69b49020f46a3fc965cf1: Status 404 returned error can't find the container with id 8ebef5b44dde7f1077abb3d2ad6020f3120127f7c5d69b49020f46a3fc965cf1
Oct 30 23:57:40.496630 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:40.496581    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g" event={"ID":"860e198e-cbb1-4cfd-b7cf-f7838bd5bb33","Type":"ContainerStarted","Data":"8ebef5b44dde7f1077abb3d2ad6020f3120127f7c5d69b49020f46a3fc965cf1"}
Oct 30 23:57:41.504232 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:41.504138    5078 generic.go:356] "Generic (PLEG): container finished" podID="860e198e-cbb1-4cfd-b7cf-f7838bd5bb33" containerID="43112cb646ecf4361eb7a2fdad5a8bbfe505d192351b5fbde662b6d3ebf013ef" exitCode=0
Oct 30 23:57:41.504697 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:41.504232    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g" event={"ID":"860e198e-cbb1-4cfd-b7cf-f7838bd5bb33","Type":"ContainerDied","Data":"43112cb646ecf4361eb7a2fdad5a8bbfe505d192351b5fbde662b6d3ebf013ef"}
Oct 30 23:57:42.640842 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:42.640818    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g"
Oct 30 23:57:42.701502 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:42.701474    5078 patch_prober.go:29] interesting pod/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:57:42.701650 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:42.701520    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q" podUID="357e99ba-4145-479f-aa06-cf148c2c52b4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:57:42.823839 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:42.823815    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-kube-api-access-8tbng\" (UniqueName: \"kubernetes.io/projected/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-kube-api-access-8tbng\") pod \"860e198e-cbb1-4cfd-b7cf-f7838bd5bb33\" (UID: \"860e198e-cbb1-4cfd-b7cf-f7838bd5bb33\") "
Oct 30 23:57:42.823994 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:42.823888    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-maas-api-xks-ca\") pod \"860e198e-cbb1-4cfd-b7cf-f7838bd5bb33\" (UID: \"860e198e-cbb1-4cfd-b7cf-f7838bd5bb33\") "
Oct 30 23:57:42.823994 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:42.823930    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-maas-api-openshift-ca\") pod \"860e198e-cbb1-4cfd-b7cf-f7838bd5bb33\" (UID: \"860e198e-cbb1-4cfd-b7cf-f7838bd5bb33\") "
Oct 30 23:57:42.824315 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:42.824292    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-maas-api-openshift-ca" pod "860e198e-cbb1-4cfd-b7cf-f7838bd5bb33" (UID: "860e198e-cbb1-4cfd-b7cf-f7838bd5bb33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:42.826026 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:42.826006    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-maas-api-xks-ca" pod "860e198e-cbb1-4cfd-b7cf-f7838bd5bb33" (UID: "860e198e-cbb1-4cfd-b7cf-f7838bd5bb33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:57:42.826132 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:42.826030    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-kube-api-access-8tbng" pod "860e198e-cbb1-4cfd-b7cf-f7838bd5bb33" (UID: "860e198e-cbb1-4cfd-b7cf-f7838bd5bb33"). InnerVolumeSpecName "kube-api-access-8tbng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:42.924560 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:42.924530    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:42.924560 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:42.924555    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:42.924560 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:42.924566    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tbng\" (UniqueName: \"kubernetes.io/projected/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33-kube-api-access-8tbng\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:43.516148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:43.516110    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g"
Oct 30 23:57:43.516345 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:43.516109    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g" event={"ID":"860e198e-cbb1-4cfd-b7cf-f7838bd5bb33","Type":"ContainerDied","Data":"8ebef5b44dde7f1077abb3d2ad6020f3120127f7c5d69b49020f46a3fc965cf1"}
Oct 30 23:57:43.516345 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:43.516274    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ebef5b44dde7f1077abb3d2ad6020f3120127f7c5d69b49020f46a3fc965cf1"
Oct 30 23:57:43.850727 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:43.850694    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj" podUID="5e4686b5-2668-4680-b215-432e012c3348" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:57:43.851199 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:43.851068    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc" podUID="ca4df19e-725c-4529-8028-18615a736d0b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:57:43.914731 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:43.914709    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:57:44.035282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.035202    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-data\") pod \"357e99ba-4145-479f-aa06-cf148c2c52b4\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") "
Oct 30 23:57:44.035282 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.035271    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-podinfo\") pod \"357e99ba-4145-479f-aa06-cf148c2c52b4\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") "
Oct 30 23:57:44.035487 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.035315    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-workload-certs\") pod \"357e99ba-4145-479f-aa06-cf148c2c52b4\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") "
Oct 30 23:57:44.035487 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.035344    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-workload-socket\") pod \"357e99ba-4145-479f-aa06-cf148c2c52b4\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") "
Oct 30 23:57:44.035487 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.035377    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357e99ba-4145-479f-aa06-cf148c2c52b4-kube-api-access-gb9xv\" (UniqueName: \"kubernetes.io/projected/357e99ba-4145-479f-aa06-cf148c2c52b4-kube-api-access-gb9xv\") pod \"357e99ba-4145-479f-aa06-cf148c2c52b4\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") "
Oct 30 23:57:44.035487 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.035415    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-envoy\") pod \"357e99ba-4145-479f-aa06-cf148c2c52b4\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") "
Oct 30 23:57:44.035487 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.035468    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-token\" (UniqueName: \"kubernetes.io/projected/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-token\") pod \"357e99ba-4145-479f-aa06-cf148c2c52b4\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") "
Oct 30 23:57:44.035738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.035504    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-data" pod "357e99ba-4145-479f-aa06-cf148c2c52b4" (UID: "357e99ba-4145-479f-aa06-cf148c2c52b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:44.035738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.035560    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/357e99ba-4145-479f-aa06-cf148c2c52b4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357e99ba-4145-479f-aa06-cf148c2c52b4-istiod-ca-cert\") pod \"357e99ba-4145-479f-aa06-cf148c2c52b4\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") "
Oct 30 23:57:44.035738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.035657    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-credential-socket\") pod \"357e99ba-4145-479f-aa06-cf148c2c52b4\" (UID: \"357e99ba-4145-479f-aa06-cf148c2c52b4\") "
Oct 30 23:57:44.035738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.035700    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-workload-socket" pod "357e99ba-4145-479f-aa06-cf148c2c52b4" (UID: "357e99ba-4145-479f-aa06-cf148c2c52b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:44.035927 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.035911    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:44.036006 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.035933    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:44.036247 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.036219    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-credential-socket" pod "357e99ba-4145-479f-aa06-cf148c2c52b4" (UID: "357e99ba-4145-479f-aa06-cf148c2c52b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:44.036521 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.036493    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-workload-certs" pod "357e99ba-4145-479f-aa06-cf148c2c52b4" (UID: "357e99ba-4145-479f-aa06-cf148c2c52b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:44.036694 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.036544    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/357e99ba-4145-479f-aa06-cf148c2c52b4-istiod-ca-cert" pod "357e99ba-4145-479f-aa06-cf148c2c52b4" (UID: "357e99ba-4145-479f-aa06-cf148c2c52b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:44.038474 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.038448    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-podinfo" pod "357e99ba-4145-479f-aa06-cf148c2c52b4" (UID: "357e99ba-4145-479f-aa06-cf148c2c52b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:57:44.038669 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.038644    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-token" pod "357e99ba-4145-479f-aa06-cf148c2c52b4" (UID: "357e99ba-4145-479f-aa06-cf148c2c52b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:44.038953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.038921    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/357e99ba-4145-479f-aa06-cf148c2c52b4-kube-api-access-gb9xv" pod "357e99ba-4145-479f-aa06-cf148c2c52b4" (UID: "357e99ba-4145-479f-aa06-cf148c2c52b4"). InnerVolumeSpecName "kube-api-access-gb9xv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:44.039026 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.038954    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-envoy" pod "357e99ba-4145-479f-aa06-cf148c2c52b4" (UID: "357e99ba-4145-479f-aa06-cf148c2c52b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:44.122500 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.122468    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"]
Oct 30 23:57:44.136482 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.136443    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:44.136482 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.136468    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:44.136482 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.136477    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb9xv\" (UniqueName: \"kubernetes.io/projected/357e99ba-4145-479f-aa06-cf148c2c52b4-kube-api-access-gb9xv\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:44.136482 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.136488    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:44.136779 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.136496    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/357e99ba-4145-479f-aa06-cf148c2c52b4-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:44.136779 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.136504    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/357e99ba-4145-479f-aa06-cf148c2c52b4-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:44.136779 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.136512    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/357e99ba-4145-479f-aa06-cf148c2c52b4-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:44.523511 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.523481    5078 generic.go:356] "Generic (PLEG): container finished" podID="357e99ba-4145-479f-aa06-cf148c2c52b4" containerID="a5601e17240d9a3e10c527feece9248f4ed46ed31f13887b782274298471a93d" exitCode=0
Oct 30 23:57:44.523687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.523541    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"
Oct 30 23:57:44.523687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.523566    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q" event={"ID":"357e99ba-4145-479f-aa06-cf148c2c52b4","Type":"ContainerDied","Data":"a5601e17240d9a3e10c527feece9248f4ed46ed31f13887b782274298471a93d"}
Oct 30 23:57:44.523687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.523635    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q" event={"ID":"357e99ba-4145-479f-aa06-cf148c2c52b4","Type":"ContainerDied","Data":"3be08087284723c18d0fd85cb2350f31a8b1c73c4efa23455ebfb416d67fe776"}
Oct 30 23:57:44.523687 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.523662    5078 scope.go:122] "RemoveContainer" containerID="a5601e17240d9a3e10c527feece9248f4ed46ed31f13887b782274298471a93d"
Oct 30 23:57:44.532535 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.532518    5078 scope.go:122] "RemoveContainer" containerID="4649efa82104f0303deeedb98930ecf6645f625e240e42b5724dee27b0dcc834"
Oct 30 23:57:44.543296 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.543273    5078 scope.go:122] "RemoveContainer" containerID="a5601e17240d9a3e10c527feece9248f4ed46ed31f13887b782274298471a93d"
Oct 30 23:57:44.543905 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:57:44.543881    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5601e17240d9a3e10c527feece9248f4ed46ed31f13887b782274298471a93d\": container with ID starting with a5601e17240d9a3e10c527feece9248f4ed46ed31f13887b782274298471a93d not found: ID does not exist" containerID="a5601e17240d9a3e10c527feece9248f4ed46ed31f13887b782274298471a93d"
Oct 30 23:57:44.543995 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.543917    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5601e17240d9a3e10c527feece9248f4ed46ed31f13887b782274298471a93d"} err="failed to get container status \"a5601e17240d9a3e10c527feece9248f4ed46ed31f13887b782274298471a93d\": rpc error: code = NotFound desc = could not find container \"a5601e17240d9a3e10c527feece9248f4ed46ed31f13887b782274298471a93d\": container with ID starting with a5601e17240d9a3e10c527feece9248f4ed46ed31f13887b782274298471a93d not found: ID does not exist"
Oct 30 23:57:44.543995 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.543944    5078 scope.go:122] "RemoveContainer" containerID="4649efa82104f0303deeedb98930ecf6645f625e240e42b5724dee27b0dcc834"
Oct 30 23:57:44.544224 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:57:44.544206    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4649efa82104f0303deeedb98930ecf6645f625e240e42b5724dee27b0dcc834\": container with ID starting with 4649efa82104f0303deeedb98930ecf6645f625e240e42b5724dee27b0dcc834 not found: ID does not exist" containerID="4649efa82104f0303deeedb98930ecf6645f625e240e42b5724dee27b0dcc834"
Oct 30 23:57:44.544264 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.544229    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4649efa82104f0303deeedb98930ecf6645f625e240e42b5724dee27b0dcc834"} err="failed to get container status \"4649efa82104f0303deeedb98930ecf6645f625e240e42b5724dee27b0dcc834\": rpc error: code = NotFound desc = could not find container \"4649efa82104f0303deeedb98930ecf6645f625e240e42b5724dee27b0dcc834\": container with ID starting with 4649efa82104f0303deeedb98930ecf6645f625e240e42b5724dee27b0dcc834 not found: ID does not exist"
Oct 30 23:57:44.689930 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.689856    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"]
Oct 30 23:57:44.739174 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:44.739142    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-333b59-openshift-default-6fd4d57b67-xn75q"]
Oct 30 23:57:46.297059 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:46.297028    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="357e99ba-4145-479f-aa06-cf148c2c52b4" path="/var/lib/kubelet/pods/357e99ba-4145-479f-aa06-cf148c2c52b4/volumes"
Oct 30 23:57:47.591153 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:57:47.591119    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:57:48.877191 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:48.877157    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7" podUID="2a6cb384-8d9a-4b7d-a936-b8c556d30170" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 23:57:49.116978 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.116954    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc"
Oct 30 23:57:49.182996 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.182964    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca4df19e-725c-4529-8028-18615a736d0b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4df19e-725c-4529-8028-18615a736d0b-plugins-config-volume\") pod \"ca4df19e-725c-4529-8028-18615a736d0b\" (UID: \"ca4df19e-725c-4529-8028-18615a736d0b\") "
Oct 30 23:57:49.183163 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.183082    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca4df19e-725c-4529-8028-18615a736d0b-kube-api-access-t64nb\" (UniqueName: \"kubernetes.io/projected/ca4df19e-725c-4529-8028-18615a736d0b-kube-api-access-t64nb\") pod \"ca4df19e-725c-4529-8028-18615a736d0b\" (UID: \"ca4df19e-725c-4529-8028-18615a736d0b\") "
Oct 30 23:57:49.183360 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.183339    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4df19e-725c-4529-8028-18615a736d0b-plugins-config-volume" pod "ca4df19e-725c-4529-8028-18615a736d0b" (UID: "ca4df19e-725c-4529-8028-18615a736d0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:49.185632 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.185585    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4df19e-725c-4529-8028-18615a736d0b-kube-api-access-t64nb" pod "ca4df19e-725c-4529-8028-18615a736d0b" (UID: "ca4df19e-725c-4529-8028-18615a736d0b"). InnerVolumeSpecName "kube-api-access-t64nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:49.213268 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.213241    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj"
Oct 30 23:57:49.283866 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.283834    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e4686b5-2668-4680-b215-432e012c3348-kube-api-access-kxhwh\" (UniqueName: \"kubernetes.io/projected/5e4686b5-2668-4680-b215-432e012c3348-kube-api-access-kxhwh\") pod \"5e4686b5-2668-4680-b215-432e012c3348\" (UID: \"5e4686b5-2668-4680-b215-432e012c3348\") "
Oct 30 23:57:49.284016 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.283880    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e4686b5-2668-4680-b215-432e012c3348-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4686b5-2668-4680-b215-432e012c3348-plugins-config-volume\") pod \"5e4686b5-2668-4680-b215-432e012c3348\" (UID: \"5e4686b5-2668-4680-b215-432e012c3348\") "
Oct 30 23:57:49.284154 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.284135    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4df19e-725c-4529-8028-18615a736d0b-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:49.284204 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.284162    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t64nb\" (UniqueName: \"kubernetes.io/projected/ca4df19e-725c-4529-8028-18615a736d0b-kube-api-access-t64nb\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:49.284250 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.284217    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e4686b5-2668-4680-b215-432e012c3348-plugins-config-volume" pod "5e4686b5-2668-4680-b215-432e012c3348" (UID: "5e4686b5-2668-4680-b215-432e012c3348"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:49.286175 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.286154    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e4686b5-2668-4680-b215-432e012c3348-kube-api-access-kxhwh" pod "5e4686b5-2668-4680-b215-432e012c3348" (UID: "5e4686b5-2668-4680-b215-432e012c3348"). InnerVolumeSpecName "kube-api-access-kxhwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:49.385148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.385113    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxhwh\" (UniqueName: \"kubernetes.io/projected/5e4686b5-2668-4680-b215-432e012c3348-kube-api-access-kxhwh\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:49.385148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.385142    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e4686b5-2668-4680-b215-432e012c3348-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:49.552780 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.552747    5078 generic.go:356] "Generic (PLEG): container finished" podID="ca4df19e-725c-4529-8028-18615a736d0b" containerID="1feaad350f7a51b88115a1cf473807e516ff1ea7864fcb218093b67b4b7328bd" exitCode=0
Oct 30 23:57:49.552949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.552799    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc"
Oct 30 23:57:49.552949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.552823    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc" event={"ID":"ca4df19e-725c-4529-8028-18615a736d0b","Type":"ContainerDied","Data":"1feaad350f7a51b88115a1cf473807e516ff1ea7864fcb218093b67b4b7328bd"}
Oct 30 23:57:49.552949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.552868    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc" event={"ID":"ca4df19e-725c-4529-8028-18615a736d0b","Type":"ContainerDied","Data":"46ebc4fc54296cdbf6be9c08784131bd93be9d0aa21d67022c1d4d98725abaa3"}
Oct 30 23:57:49.552949 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.552892    5078 scope.go:122] "RemoveContainer" containerID="1feaad350f7a51b88115a1cf473807e516ff1ea7864fcb218093b67b4b7328bd"
Oct 30 23:57:49.554343 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.554316    5078 generic.go:356] "Generic (PLEG): container finished" podID="5e4686b5-2668-4680-b215-432e012c3348" containerID="a2d3c35f37b9d2e3fb6607c626cbf188585434e08d8f5bd7d49def7d68587fe5" exitCode=0
Oct 30 23:57:49.554476 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.554358    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj"
Oct 30 23:57:49.554476 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.554410    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj" event={"ID":"5e4686b5-2668-4680-b215-432e012c3348","Type":"ContainerDied","Data":"a2d3c35f37b9d2e3fb6607c626cbf188585434e08d8f5bd7d49def7d68587fe5"}
Oct 30 23:57:49.554476 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.554460    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj" event={"ID":"5e4686b5-2668-4680-b215-432e012c3348","Type":"ContainerDied","Data":"772667bf05d7cf5a2fe8dd7b507ea2babb1ad0ecfaa1a26fba3287e351c75703"}
Oct 30 23:57:49.563276 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.563259    5078 scope.go:122] "RemoveContainer" containerID="1feaad350f7a51b88115a1cf473807e516ff1ea7864fcb218093b67b4b7328bd"
Oct 30 23:57:49.563701 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:57:49.563674    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1feaad350f7a51b88115a1cf473807e516ff1ea7864fcb218093b67b4b7328bd\": container with ID starting with 1feaad350f7a51b88115a1cf473807e516ff1ea7864fcb218093b67b4b7328bd not found: ID does not exist" containerID="1feaad350f7a51b88115a1cf473807e516ff1ea7864fcb218093b67b4b7328bd"
Oct 30 23:57:49.563793 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.563711    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1feaad350f7a51b88115a1cf473807e516ff1ea7864fcb218093b67b4b7328bd"} err="failed to get container status \"1feaad350f7a51b88115a1cf473807e516ff1ea7864fcb218093b67b4b7328bd\": rpc error: code = NotFound desc = could not find container \"1feaad350f7a51b88115a1cf473807e516ff1ea7864fcb218093b67b4b7328bd\": container with ID starting with 1feaad350f7a51b88115a1cf473807e516ff1ea7864fcb218093b67b4b7328bd not found: ID does not exist"
Oct 30 23:57:49.563793 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.563733    5078 scope.go:122] "RemoveContainer" containerID="a2d3c35f37b9d2e3fb6607c626cbf188585434e08d8f5bd7d49def7d68587fe5"
Oct 30 23:57:49.571816 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.571798    5078 scope.go:122] "RemoveContainer" containerID="a2d3c35f37b9d2e3fb6607c626cbf188585434e08d8f5bd7d49def7d68587fe5"
Oct 30 23:57:49.572050 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:57:49.572033    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d3c35f37b9d2e3fb6607c626cbf188585434e08d8f5bd7d49def7d68587fe5\": container with ID starting with a2d3c35f37b9d2e3fb6607c626cbf188585434e08d8f5bd7d49def7d68587fe5 not found: ID does not exist" containerID="a2d3c35f37b9d2e3fb6607c626cbf188585434e08d8f5bd7d49def7d68587fe5"
Oct 30 23:57:49.572095 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.572059    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d3c35f37b9d2e3fb6607c626cbf188585434e08d8f5bd7d49def7d68587fe5"} err="failed to get container status \"a2d3c35f37b9d2e3fb6607c626cbf188585434e08d8f5bd7d49def7d68587fe5\": rpc error: code = NotFound desc = could not find container \"a2d3c35f37b9d2e3fb6607c626cbf188585434e08d8f5bd7d49def7d68587fe5\": container with ID starting with a2d3c35f37b9d2e3fb6607c626cbf188585434e08d8f5bd7d49def7d68587fe5 not found: ID does not exist"
Oct 30 23:57:49.717203 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.717177    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc"]
Oct 30 23:57:49.721535 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.721513    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-5dc3c0-fcfbcb777-5m6qc"]
Oct 30 23:57:49.831224 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.831156    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj"]
Oct 30 23:57:49.837912 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:49.837889    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-5dc3c0-59f6bdc479-nhzjj"]
Oct 30 23:57:50.296140 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:50.296111    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e4686b5-2668-4680-b215-432e012c3348" path="/var/lib/kubelet/pods/5e4686b5-2668-4680-b215-432e012c3348/volumes"
Oct 30 23:57:50.296519 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:50.296442    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca4df19e-725c-4529-8028-18615a736d0b" path="/var/lib/kubelet/pods/ca4df19e-725c-4529-8028-18615a736d0b/volumes"
Oct 30 23:57:52.946844 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:52.946819    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7"
Oct 30 23:57:53.013148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:53.013071    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a6cb384-8d9a-4b7d-a936-b8c556d30170-kube-api-access-kbhnk\" (UniqueName: \"kubernetes.io/projected/2a6cb384-8d9a-4b7d-a936-b8c556d30170-kube-api-access-kbhnk\") pod \"2a6cb384-8d9a-4b7d-a936-b8c556d30170\" (UID: \"2a6cb384-8d9a-4b7d-a936-b8c556d30170\") "
Oct 30 23:57:53.013148 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:53.013118    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a6cb384-8d9a-4b7d-a936-b8c556d30170-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6cb384-8d9a-4b7d-a936-b8c556d30170-plugins-config-volume\") pod \"2a6cb384-8d9a-4b7d-a936-b8c556d30170\" (UID: \"2a6cb384-8d9a-4b7d-a936-b8c556d30170\") "
Oct 30 23:57:53.013532 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:53.013506    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a6cb384-8d9a-4b7d-a936-b8c556d30170-plugins-config-volume" pod "2a6cb384-8d9a-4b7d-a936-b8c556d30170" (UID: "2a6cb384-8d9a-4b7d-a936-b8c556d30170"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:57:53.015302 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:53.015281    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6cb384-8d9a-4b7d-a936-b8c556d30170-kube-api-access-kbhnk" pod "2a6cb384-8d9a-4b7d-a936-b8c556d30170" (UID: "2a6cb384-8d9a-4b7d-a936-b8c556d30170"). InnerVolumeSpecName "kube-api-access-kbhnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:53.114066 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:53.114032    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbhnk\" (UniqueName: \"kubernetes.io/projected/2a6cb384-8d9a-4b7d-a936-b8c556d30170-kube-api-access-kbhnk\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:53.114066 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:53.114060    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a6cb384-8d9a-4b7d-a936-b8c556d30170-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:53.587883 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:53.587843    5078 generic.go:356] "Generic (PLEG): container finished" podID="2a6cb384-8d9a-4b7d-a936-b8c556d30170" containerID="6978d4e0aafebfc7c1913d57309fe7e93fda6e19b2e996c3829b41d82bff56a8" exitCode=0
Oct 30 23:57:53.588046 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:53.587904    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7"
Oct 30 23:57:53.588046 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:53.587924    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7" event={"ID":"2a6cb384-8d9a-4b7d-a936-b8c556d30170","Type":"ContainerDied","Data":"6978d4e0aafebfc7c1913d57309fe7e93fda6e19b2e996c3829b41d82bff56a8"}
Oct 30 23:57:53.588046 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:53.587998    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7" event={"ID":"2a6cb384-8d9a-4b7d-a936-b8c556d30170","Type":"ContainerDied","Data":"735bad881a4e5f85dd359dd87086f22d18da1fce78357093e00b58d0f6a8cd5f"}
Oct 30 23:57:53.588046 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:53.588020    5078 scope.go:122] "RemoveContainer" containerID="6978d4e0aafebfc7c1913d57309fe7e93fda6e19b2e996c3829b41d82bff56a8"
Oct 30 23:57:53.597004 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:53.596977    5078 scope.go:122] "RemoveContainer" containerID="6978d4e0aafebfc7c1913d57309fe7e93fda6e19b2e996c3829b41d82bff56a8"
Oct 30 23:57:53.597283 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:57:53.597264    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6978d4e0aafebfc7c1913d57309fe7e93fda6e19b2e996c3829b41d82bff56a8\": container with ID starting with 6978d4e0aafebfc7c1913d57309fe7e93fda6e19b2e996c3829b41d82bff56a8 not found: ID does not exist" containerID="6978d4e0aafebfc7c1913d57309fe7e93fda6e19b2e996c3829b41d82bff56a8"
Oct 30 23:57:53.597364 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:53.597291    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6978d4e0aafebfc7c1913d57309fe7e93fda6e19b2e996c3829b41d82bff56a8"} err="failed to get container status \"6978d4e0aafebfc7c1913d57309fe7e93fda6e19b2e996c3829b41d82bff56a8\": rpc error: code = NotFound desc = could not find container \"6978d4e0aafebfc7c1913d57309fe7e93fda6e19b2e996c3829b41d82bff56a8\": container with ID starting with 6978d4e0aafebfc7c1913d57309fe7e93fda6e19b2e996c3829b41d82bff56a8 not found: ID does not exist"
Oct 30 23:57:53.786396 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:53.786360    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7"]
Oct 30 23:57:53.835690 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:53.835658    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-aadaa6-6fc8879f75-xnks7"]
Oct 30 23:57:54.296050 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:54.296020    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a6cb384-8d9a-4b7d-a936-b8c556d30170" path="/var/lib/kubelet/pods/2a6cb384-8d9a-4b7d-a936-b8c556d30170/volumes"
Oct 30 23:57:54.780813 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:54.780738    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt"]
Oct 30 23:57:54.830624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:54.830578    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-aadaa6-6n5mt"]
Oct 30 23:57:56.297383 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:56.297354    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7f1998f-5d7a-493b-bf98-30de70ac25b7" path="/var/lib/kubelet/pods/a7f1998f-5d7a-493b-bf98-30de70ac25b7/volumes"
Oct 30 23:57:59.142568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.142525    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main" containerID="cri-o://056d2f7a98b41e214d7be1c931ed645aeb8a0863ad1d8b10cea0d00b2929bbbe" gracePeriod=45
Oct 30 23:57:59.379032 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.379008    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:57:59.389356 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389329    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2"]
Oct 30 23:57:59.389649 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389636    5078 cpu_manager.go:414] "containerMap: removing container" podUID="5e4686b5-2668-4680-b215-432e012c3348" containerName="payload-pre-processing"
Oct 30 23:57:59.389707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389651    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e4686b5-2668-4680-b215-432e012c3348" containerName="payload-pre-processing"
Oct 30 23:57:59.389707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389669    5078 cpu_manager.go:414] "containerMap: removing container" podUID="2a6cb384-8d9a-4b7d-a936-b8c556d30170" containerName="payload-processing"
Oct 30 23:57:59.389707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389674    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a6cb384-8d9a-4b7d-a936-b8c556d30170" containerName="payload-processing"
Oct 30 23:57:59.389707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389680    5078 cpu_manager.go:414] "containerMap: removing container" podUID="860e198e-cbb1-4cfd-b7cf-f7838bd5bb33" containerName="revoke-keys"
Oct 30 23:57:59.389707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389685    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860e198e-cbb1-4cfd-b7cf-f7838bd5bb33" containerName="revoke-keys"
Oct 30 23:57:59.389707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389699    5078 cpu_manager.go:414] "containerMap: removing container" podUID="357e99ba-4145-479f-aa06-cf148c2c52b4" containerName="istio-proxy"
Oct 30 23:57:59.389707 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389704    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357e99ba-4145-479f-aa06-cf148c2c52b4" containerName="istio-proxy"
Oct 30 23:57:59.389931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389715    5078 cpu_manager.go:414] "containerMap: removing container" podUID="357e99ba-4145-479f-aa06-cf148c2c52b4" containerName="istio-proxy"
Oct 30 23:57:59.389931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389720    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="357e99ba-4145-479f-aa06-cf148c2c52b4" containerName="istio-proxy"
Oct 30 23:57:59.389931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389727    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ca4df19e-725c-4529-8028-18615a736d0b" containerName="payload-processing"
Oct 30 23:57:59.389931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389731    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca4df19e-725c-4529-8028-18615a736d0b" containerName="payload-processing"
Oct 30 23:57:59.389931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389738    5078 cpu_manager.go:414] "containerMap: removing container" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main"
Oct 30 23:57:59.389931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389742    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main"
Oct 30 23:57:59.389931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389788    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e4686b5-2668-4680-b215-432e012c3348" containerName="payload-pre-processing"
Oct 30 23:57:59.389931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389799    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a6cb384-8d9a-4b7d-a936-b8c556d30170" containerName="payload-processing"
Oct 30 23:57:59.389931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389805    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="860e198e-cbb1-4cfd-b7cf-f7838bd5bb33" containerName="revoke-keys"
Oct 30 23:57:59.389931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389813    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="357e99ba-4145-479f-aa06-cf148c2c52b4" containerName="istio-proxy"
Oct 30 23:57:59.389931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389819    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="357e99ba-4145-479f-aa06-cf148c2c52b4" containerName="istio-proxy"
Oct 30 23:57:59.389931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389823    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca4df19e-725c-4529-8028-18615a736d0b" containerName="payload-processing"
Oct 30 23:57:59.389931 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.389829    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerName="main"
Oct 30 23:57:59.392671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.392628    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2"
Oct 30 23:57:59.454591 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.454564    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2"]
Oct 30 23:57:59.469025 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.468997    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-tls-certs\" (UniqueName: \"kubernetes.io/secret/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-tls-certs\") pod \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") "
Oct 30 23:57:59.469168 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.469042    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-kube-api-access-zt45b\" (UniqueName: \"kubernetes.io/projected/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-kube-api-access-zt45b\") pod \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") "
Oct 30 23:57:59.469168 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.469093    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-dshm\") pod \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") "
Oct 30 23:57:59.469168 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.469142    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-home\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-home\") pod \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") "
Oct 30 23:57:59.469378 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.469214    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-model-cache\") pod \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") "
Oct 30 23:57:59.469378 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.469261    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-tmp-dir\") pod \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\" (UID: \"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2\") "
Oct 30 23:57:59.469496 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.469408    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534c4672-90ed-47ce-9a7b-28aac0ccc886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bea43a4-bgmp2\" (UID: \"534c4672-90ed-47ce-9a7b-28aac0ccc886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2"
Oct 30 23:57:59.469496 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.469460    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phlwp\" (UniqueName: \"kubernetes.io/projected/534c4672-90ed-47ce-9a7b-28aac0ccc886-kube-api-access-phlwp\") pod \"maas-api-revoke-sub-3bea43a4-bgmp2\" (UID: \"534c4672-90ed-47ce-9a7b-28aac0ccc886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2"
Oct 30 23:57:59.469496 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.469466    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-model-cache" pod "a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" (UID: "a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:59.469701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.469571    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534c4672-90ed-47ce-9a7b-28aac0ccc886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bea43a4-bgmp2\" (UID: \"534c4672-90ed-47ce-9a7b-28aac0ccc886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2"
Oct 30 23:57:59.469701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.469635    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:59.469701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.469643    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-tmp-dir" pod "a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" (UID: "a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:59.470032 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.469710    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-home" pod "a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" (UID: "a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:59.471653 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.471628    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-kube-api-access-zt45b" pod "a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" (UID: "a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2"). InnerVolumeSpecName "kube-api-access-zt45b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:57:59.471779 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.471704    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-tls-certs" pod "a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" (UID: "a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:57:59.471855 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.471832    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-dshm" pod "a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" (UID: "a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:57:59.481187 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.481163    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:57:59.481187 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.481165    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:57:59.481357 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.481205    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:57:59.481357 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.481239    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:57:59.570675 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.570640    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534c4672-90ed-47ce-9a7b-28aac0ccc886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bea43a4-bgmp2\" (UID: \"534c4672-90ed-47ce-9a7b-28aac0ccc886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2"
Oct 30 23:57:59.570840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.570700    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phlwp\" (UniqueName: \"kubernetes.io/projected/534c4672-90ed-47ce-9a7b-28aac0ccc886-kube-api-access-phlwp\") pod \"maas-api-revoke-sub-3bea43a4-bgmp2\" (UID: \"534c4672-90ed-47ce-9a7b-28aac0ccc886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2"
Oct 30 23:57:59.570840 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.570771    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534c4672-90ed-47ce-9a7b-28aac0ccc886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bea43a4-bgmp2\" (UID: \"534c4672-90ed-47ce-9a7b-28aac0ccc886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2"
Oct 30 23:57:59.570966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.570865    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:59.570966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.570883    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zt45b\" (UniqueName: \"kubernetes.io/projected/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-kube-api-access-zt45b\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:59.570966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.570899    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:59.570966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.570913    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:59.570966 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.570927    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:57:59.571451 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.571424    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534c4672-90ed-47ce-9a7b-28aac0ccc886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bea43a4-bgmp2\" (UID: \"534c4672-90ed-47ce-9a7b-28aac0ccc886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2"
Oct 30 23:57:59.573212 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.573193    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534c4672-90ed-47ce-9a7b-28aac0ccc886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bea43a4-bgmp2\" (UID: \"534c4672-90ed-47ce-9a7b-28aac0ccc886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2"
Oct 30 23:57:59.630649 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.630579    5078 generic.go:356] "Generic (PLEG): container finished" podID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" containerID="056d2f7a98b41e214d7be1c931ed645aeb8a0863ad1d8b10cea0d00b2929bbbe" exitCode=0
Oct 30 23:57:59.630843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.630727    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"
Oct 30 23:57:59.630843 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.630736    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" event={"ID":"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2","Type":"ContainerDied","Data":"056d2f7a98b41e214d7be1c931ed645aeb8a0863ad1d8b10cea0d00b2929bbbe"}
Oct 30 23:57:59.630937 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.630913    5078 scope.go:122] "RemoveContainer" containerID="056d2f7a98b41e214d7be1c931ed645aeb8a0863ad1d8b10cea0d00b2929bbbe"
Oct 30 23:57:59.630982 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.630923    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l" event={"ID":"a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2","Type":"ContainerDied","Data":"cd7ed027ade75ccef042943d27318eb34f5462e936b90736671689ab51c0fb7f"}
Oct 30 23:57:59.640818 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.640797    5078 scope.go:122] "RemoveContainer" containerID="056d2f7a98b41e214d7be1c931ed645aeb8a0863ad1d8b10cea0d00b2929bbbe"
Oct 30 23:57:59.641149 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:57:59.641067    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"056d2f7a98b41e214d7be1c931ed645aeb8a0863ad1d8b10cea0d00b2929bbbe\": container with ID starting with 056d2f7a98b41e214d7be1c931ed645aeb8a0863ad1d8b10cea0d00b2929bbbe not found: ID does not exist" containerID="056d2f7a98b41e214d7be1c931ed645aeb8a0863ad1d8b10cea0d00b2929bbbe"
Oct 30 23:57:59.641149 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.641092    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"056d2f7a98b41e214d7be1c931ed645aeb8a0863ad1d8b10cea0d00b2929bbbe"} err="failed to get container status \"056d2f7a98b41e214d7be1c931ed645aeb8a0863ad1d8b10cea0d00b2929bbbe\": rpc error: code = NotFound desc = could not find container \"056d2f7a98b41e214d7be1c931ed645aeb8a0863ad1d8b10cea0d00b2929bbbe\": container with ID starting with 056d2f7a98b41e214d7be1c931ed645aeb8a0863ad1d8b10cea0d00b2929bbbe not found: ID does not exist"
Oct 30 23:57:59.643741 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.643690    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phlwp\" (UniqueName: \"kubernetes.io/projected/534c4672-90ed-47ce-9a7b-28aac0ccc886-kube-api-access-phlwp\") pod \"maas-api-revoke-sub-3bea43a4-bgmp2\" (UID: \"534c4672-90ed-47ce-9a7b-28aac0ccc886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2"
Oct 30 23:57:59.705057 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.705024    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2"
Oct 30 23:57:59.755047 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.755015    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"]
Oct 30 23:57:59.759097 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:57:59.759070    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b81a7a/distinct-2-w1-b81a7a-kserve-7795cb7b4c-xck8l"]
Oct 30 23:58:00.087126 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:00.087099    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2"]
Oct 30 23:58:00.090194 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:58:00.090165    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534c4672_90ed_47ce_9a7b_28aac0ccc886.slice/crio-a5bcf13f74ee9a98f236357d143371c2a545b74cf9327f061312401440b65477 WatchSource:0}: Error finding container a5bcf13f74ee9a98f236357d143371c2a545b74cf9327f061312401440b65477: Status 404 returned error can't find the container with id a5bcf13f74ee9a98f236357d143371c2a545b74cf9327f061312401440b65477
Oct 30 23:58:00.296540 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:00.296513    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2" path="/var/lib/kubelet/pods/a0be3c19-e6ea-49f2-b4c6-2945a49fcdd2/volumes"
Oct 30 23:58:00.637424 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:00.637338    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2" event={"ID":"534c4672-90ed-47ce-9a7b-28aac0ccc886","Type":"ContainerStarted","Data":"39f46449da218ecb927d78c20ccedb2f10aca3b4875361ba8257e18930b2787c"}
Oct 30 23:58:00.637424 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:00.637394    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2" event={"ID":"534c4672-90ed-47ce-9a7b-28aac0ccc886","Type":"ContainerStarted","Data":"a5bcf13f74ee9a98f236357d143371c2a545b74cf9327f061312401440b65477"}
Oct 30 23:58:00.755233 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:00.755184    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2" podStartSLOduration=1.755169996 podStartE2EDuration="1.755169996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:00.754496508 +0000 UTC m=+3066.642402861" watchObservedRunningTime="2026-10-30 23:58:00.755169996 +0000 UTC m=+3066.643076348"
Oct 30 23:58:01.645903 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:01.645868    5078 generic.go:356] "Generic (PLEG): container finished" podID="534c4672-90ed-47ce-9a7b-28aac0ccc886" containerID="39f46449da218ecb927d78c20ccedb2f10aca3b4875361ba8257e18930b2787c" exitCode=0
Oct 30 23:58:01.646327 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:01.645965    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2" event={"ID":"534c4672-90ed-47ce-9a7b-28aac0ccc886","Type":"ContainerDied","Data":"39f46449da218ecb927d78c20ccedb2f10aca3b4875361ba8257e18930b2787c"}
Oct 30 23:58:01.743944 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:01.743906    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"]
Oct 30 23:58:01.744189 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:01.744165    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr" podUID="dbde5115-111a-42ac-a735-8be11caf2de6" containerName="istio-proxy" containerID="cri-o://25fd3e627eb88364728578d1997ad5f7a1aad39212362784aaf16d7ee2a9bd68" gracePeriod=30
Oct 30 23:58:02.776354 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:02.776326    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2"
Oct 30 23:58:02.898846 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:02.898805    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/534c4672-90ed-47ce-9a7b-28aac0ccc886-kube-api-access-phlwp\" (UniqueName: \"kubernetes.io/projected/534c4672-90ed-47ce-9a7b-28aac0ccc886-kube-api-access-phlwp\") pod \"534c4672-90ed-47ce-9a7b-28aac0ccc886\" (UID: \"534c4672-90ed-47ce-9a7b-28aac0ccc886\") "
Oct 30 23:58:02.898846 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:02.898842    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/534c4672-90ed-47ce-9a7b-28aac0ccc886-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534c4672-90ed-47ce-9a7b-28aac0ccc886-maas-api-openshift-ca\") pod \"534c4672-90ed-47ce-9a7b-28aac0ccc886\" (UID: \"534c4672-90ed-47ce-9a7b-28aac0ccc886\") "
Oct 30 23:58:02.899091 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:02.898879    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/534c4672-90ed-47ce-9a7b-28aac0ccc886-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534c4672-90ed-47ce-9a7b-28aac0ccc886-maas-api-xks-ca\") pod \"534c4672-90ed-47ce-9a7b-28aac0ccc886\" (UID: \"534c4672-90ed-47ce-9a7b-28aac0ccc886\") "
Oct 30 23:58:02.899260 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:02.899233    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/534c4672-90ed-47ce-9a7b-28aac0ccc886-maas-api-openshift-ca" pod "534c4672-90ed-47ce-9a7b-28aac0ccc886" (UID: "534c4672-90ed-47ce-9a7b-28aac0ccc886"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:02.901227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:02.901199    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/534c4672-90ed-47ce-9a7b-28aac0ccc886-maas-api-xks-ca" pod "534c4672-90ed-47ce-9a7b-28aac0ccc886" (UID: "534c4672-90ed-47ce-9a7b-28aac0ccc886"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:02.901227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:02.901207    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534c4672-90ed-47ce-9a7b-28aac0ccc886-kube-api-access-phlwp" pod "534c4672-90ed-47ce-9a7b-28aac0ccc886" (UID: "534c4672-90ed-47ce-9a7b-28aac0ccc886"). InnerVolumeSpecName "kube-api-access-phlwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:02.999953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:02.999923    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phlwp\" (UniqueName: \"kubernetes.io/projected/534c4672-90ed-47ce-9a7b-28aac0ccc886-kube-api-access-phlwp\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:02.999953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:02.999954    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534c4672-90ed-47ce-9a7b-28aac0ccc886-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:02.999953 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:02.999974    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534c4672-90ed-47ce-9a7b-28aac0ccc886-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:03.657783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:03.657757    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2"
Oct 30 23:58:03.657783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:03.657777    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2" event={"ID":"534c4672-90ed-47ce-9a7b-28aac0ccc886","Type":"ContainerDied","Data":"a5bcf13f74ee9a98f236357d143371c2a545b74cf9327f061312401440b65477"}
Oct 30 23:58:03.657992 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:03.657818    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5bcf13f74ee9a98f236357d143371c2a545b74cf9327f061312401440b65477"
Oct 30 23:58:03.839663 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:03.839632    5078 scope.go:122] "RemoveContainer" containerID="5ea64bce9a9cdb0bab67abe5e3d6ef1d19a0a5b8f9a5fd006fb5312e68409372"
Oct 30 23:58:06.998532 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:06.998511    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:58:07.033686 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.033630    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbde5115-111a-42ac-a735-8be11caf2de6-kube-api-access-qf796\" (UniqueName: \"kubernetes.io/projected/dbde5115-111a-42ac-a735-8be11caf2de6-kube-api-access-qf796\") pod \"dbde5115-111a-42ac-a735-8be11caf2de6\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") "
Oct 30 23:58:07.033686 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.033667    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbde5115-111a-42ac-a735-8be11caf2de6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbde5115-111a-42ac-a735-8be11caf2de6-istiod-ca-cert\") pod \"dbde5115-111a-42ac-a735-8be11caf2de6\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") "
Oct 30 23:58:07.033895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.033697    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbde5115-111a-42ac-a735-8be11caf2de6-istio-token\" (UniqueName: \"kubernetes.io/projected/dbde5115-111a-42ac-a735-8be11caf2de6-istio-token\") pod \"dbde5115-111a-42ac-a735-8be11caf2de6\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") "
Oct 30 23:58:07.033895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.033727    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-istio-envoy\") pod \"dbde5115-111a-42ac-a735-8be11caf2de6\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") "
Oct 30 23:58:07.033895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.033754    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbde5115-111a-42ac-a735-8be11caf2de6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbde5115-111a-42ac-a735-8be11caf2de6-istio-podinfo\") pod \"dbde5115-111a-42ac-a735-8be11caf2de6\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") "
Oct 30 23:58:07.033895 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.033795    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-workload-socket\") pod \"dbde5115-111a-42ac-a735-8be11caf2de6\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") "
Oct 30 23:58:07.034137 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.033914    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-credential-socket\") pod \"dbde5115-111a-42ac-a735-8be11caf2de6\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") "
Oct 30 23:58:07.034137 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.033963    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-istio-data\") pod \"dbde5115-111a-42ac-a735-8be11caf2de6\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") "
Oct 30 23:58:07.034137 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.034004    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-workload-certs\") pod \"dbde5115-111a-42ac-a735-8be11caf2de6\" (UID: \"dbde5115-111a-42ac-a735-8be11caf2de6\") "
Oct 30 23:58:07.034137 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.034078    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-workload-socket" pod "dbde5115-111a-42ac-a735-8be11caf2de6" (UID: "dbde5115-111a-42ac-a735-8be11caf2de6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:07.034137 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.034087    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbde5115-111a-42ac-a735-8be11caf2de6-istiod-ca-cert" pod "dbde5115-111a-42ac-a735-8be11caf2de6" (UID: "dbde5115-111a-42ac-a735-8be11caf2de6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:07.034399 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.034273    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-credential-socket" pod "dbde5115-111a-42ac-a735-8be11caf2de6" (UID: "dbde5115-111a-42ac-a735-8be11caf2de6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:07.034399 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.034288    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-istio-data" pod "dbde5115-111a-42ac-a735-8be11caf2de6" (UID: "dbde5115-111a-42ac-a735-8be11caf2de6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:07.034399 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.034327    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbde5115-111a-42ac-a735-8be11caf2de6-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:07.034399 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.034349    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:07.034680 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.034661    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-workload-certs" pod "dbde5115-111a-42ac-a735-8be11caf2de6" (UID: "dbde5115-111a-42ac-a735-8be11caf2de6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:07.036002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.035971    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbde5115-111a-42ac-a735-8be11caf2de6-istio-podinfo" pod "dbde5115-111a-42ac-a735-8be11caf2de6" (UID: "dbde5115-111a-42ac-a735-8be11caf2de6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:58:07.036092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.036037    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-istio-envoy" pod "dbde5115-111a-42ac-a735-8be11caf2de6" (UID: "dbde5115-111a-42ac-a735-8be11caf2de6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:07.036489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.036468    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbde5115-111a-42ac-a735-8be11caf2de6-kube-api-access-qf796" pod "dbde5115-111a-42ac-a735-8be11caf2de6" (UID: "dbde5115-111a-42ac-a735-8be11caf2de6"). InnerVolumeSpecName "kube-api-access-qf796". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:07.036489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.036474    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbde5115-111a-42ac-a735-8be11caf2de6-istio-token" pod "dbde5115-111a-42ac-a735-8be11caf2de6" (UID: "dbde5115-111a-42ac-a735-8be11caf2de6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:07.135318 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.135294    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf796\" (UniqueName: \"kubernetes.io/projected/dbde5115-111a-42ac-a735-8be11caf2de6-kube-api-access-qf796\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:07.135318 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.135318    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbde5115-111a-42ac-a735-8be11caf2de6-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:07.135484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.135328    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:07.135484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.135337    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbde5115-111a-42ac-a735-8be11caf2de6-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:07.135484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.135348    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:07.135484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.135356    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:07.135484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.135364    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbde5115-111a-42ac-a735-8be11caf2de6-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:07.678515 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.678486    5078 generic.go:356] "Generic (PLEG): container finished" podID="dbde5115-111a-42ac-a735-8be11caf2de6" containerID="25fd3e627eb88364728578d1997ad5f7a1aad39212362784aaf16d7ee2a9bd68" exitCode=0
Oct 30 23:58:07.678705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.678540    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"
Oct 30 23:58:07.678705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.678574    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr" event={"ID":"dbde5115-111a-42ac-a735-8be11caf2de6","Type":"ContainerDied","Data":"25fd3e627eb88364728578d1997ad5f7a1aad39212362784aaf16d7ee2a9bd68"}
Oct 30 23:58:07.678705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.678642    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr" event={"ID":"dbde5115-111a-42ac-a735-8be11caf2de6","Type":"ContainerDied","Data":"4ae9043c8ca15498a7806074dd04234e70d73a634ae31041cc9a113b2bced86b"}
Oct 30 23:58:07.678705 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.678666    5078 scope.go:122] "RemoveContainer" containerID="25fd3e627eb88364728578d1997ad5f7a1aad39212362784aaf16d7ee2a9bd68"
Oct 30 23:58:07.688021 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.688002    5078 scope.go:122] "RemoveContainer" containerID="fa2fff8d7432e3ecb51927a8ca1dfac7ea006c47a0069719253ee440e02fd8ea"
Oct 30 23:58:07.695745 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.695732    5078 scope.go:122] "RemoveContainer" containerID="25fd3e627eb88364728578d1997ad5f7a1aad39212362784aaf16d7ee2a9bd68"
Oct 30 23:58:07.695992 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:58:07.695975    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25fd3e627eb88364728578d1997ad5f7a1aad39212362784aaf16d7ee2a9bd68\": container with ID starting with 25fd3e627eb88364728578d1997ad5f7a1aad39212362784aaf16d7ee2a9bd68 not found: ID does not exist" containerID="25fd3e627eb88364728578d1997ad5f7a1aad39212362784aaf16d7ee2a9bd68"
Oct 30 23:58:07.696029 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.696002    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fd3e627eb88364728578d1997ad5f7a1aad39212362784aaf16d7ee2a9bd68"} err="failed to get container status \"25fd3e627eb88364728578d1997ad5f7a1aad39212362784aaf16d7ee2a9bd68\": rpc error: code = NotFound desc = could not find container \"25fd3e627eb88364728578d1997ad5f7a1aad39212362784aaf16d7ee2a9bd68\": container with ID starting with 25fd3e627eb88364728578d1997ad5f7a1aad39212362784aaf16d7ee2a9bd68 not found: ID does not exist"
Oct 30 23:58:07.696029 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.696021    5078 scope.go:122] "RemoveContainer" containerID="fa2fff8d7432e3ecb51927a8ca1dfac7ea006c47a0069719253ee440e02fd8ea"
Oct 30 23:58:07.696238 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:58:07.696224    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2fff8d7432e3ecb51927a8ca1dfac7ea006c47a0069719253ee440e02fd8ea\": container with ID starting with fa2fff8d7432e3ecb51927a8ca1dfac7ea006c47a0069719253ee440e02fd8ea not found: ID does not exist" containerID="fa2fff8d7432e3ecb51927a8ca1dfac7ea006c47a0069719253ee440e02fd8ea"
Oct 30 23:58:07.696276 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.696243    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2fff8d7432e3ecb51927a8ca1dfac7ea006c47a0069719253ee440e02fd8ea"} err="failed to get container status \"fa2fff8d7432e3ecb51927a8ca1dfac7ea006c47a0069719253ee440e02fd8ea\": rpc error: code = NotFound desc = could not find container \"fa2fff8d7432e3ecb51927a8ca1dfac7ea006c47a0069719253ee440e02fd8ea\": container with ID starting with fa2fff8d7432e3ecb51927a8ca1dfac7ea006c47a0069719253ee440e02fd8ea not found: ID does not exist"
Oct 30 23:58:07.841768 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.841733    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"]
Oct 30 23:58:07.891625 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:07.891580    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-5dc3c0-openshift-default-5d4456f8c9-lvlcr"]
Oct 30 23:58:08.296368 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:08.296333    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbde5115-111a-42ac-a735-8be11caf2de6" path="/var/lib/kubelet/pods/dbde5115-111a-42ac-a735-8be11caf2de6/volumes"
Oct 30 23:58:16.827385 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:16.827347    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g"]
Oct 30 23:58:16.877170 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:16.877135    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-b81a7a-mm55g"]
Oct 30 23:58:17.590381 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:58:17.590346    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:58:18.295555 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:18.295510    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="860e198e-cbb1-4cfd-b7cf-f7838bd5bb33" path="/var/lib/kubelet/pods/860e198e-cbb1-4cfd-b7cf-f7838bd5bb33/volumes"
Oct 30 23:58:23.472816 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:23.472783    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"]
Oct 30 23:58:23.473266 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:23.472976    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p" podUID="e88bd702-4a73-4443-9dce-eed8bde3c933" containerName="istio-proxy" containerID="cri-o://115499743e8d570c40a55ce0df5bfda246b58bf7fe02690f7b42e1953e42f9e0" gracePeriod=30
Oct 30 23:58:24.493858 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:24.493826    5078 patch_prober.go:29] interesting pod/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 23:58:24.494244 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:24.493889    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p" podUID="e88bd702-4a73-4443-9dce-eed8bde3c933" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 23:58:28.711478 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.711449    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:58:28.776848 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.776769    5078 generic.go:356] "Generic (PLEG): container finished" podID="e88bd702-4a73-4443-9dce-eed8bde3c933" containerID="115499743e8d570c40a55ce0df5bfda246b58bf7fe02690f7b42e1953e42f9e0" exitCode=0
Oct 30 23:58:28.776848 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.776818    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"
Oct 30 23:58:28.777091 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.776855    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p" event={"ID":"e88bd702-4a73-4443-9dce-eed8bde3c933","Type":"ContainerDied","Data":"115499743e8d570c40a55ce0df5bfda246b58bf7fe02690f7b42e1953e42f9e0"}
Oct 30 23:58:28.777091 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.776919    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p" event={"ID":"e88bd702-4a73-4443-9dce-eed8bde3c933","Type":"ContainerDied","Data":"aef5c33a26f2ce3c1c18347285d233b4838866c8068c5ecee1e2edd32db7c78a"}
Oct 30 23:58:28.777091 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.776950    5078 scope.go:122] "RemoveContainer" containerID="115499743e8d570c40a55ce0df5bfda246b58bf7fe02690f7b42e1953e42f9e0"
Oct 30 23:58:28.785494 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.785475    5078 scope.go:122] "RemoveContainer" containerID="115499743e8d570c40a55ce0df5bfda246b58bf7fe02690f7b42e1953e42f9e0"
Oct 30 23:58:28.785776 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:58:28.785758    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115499743e8d570c40a55ce0df5bfda246b58bf7fe02690f7b42e1953e42f9e0\": container with ID starting with 115499743e8d570c40a55ce0df5bfda246b58bf7fe02690f7b42e1953e42f9e0 not found: ID does not exist" containerID="115499743e8d570c40a55ce0df5bfda246b58bf7fe02690f7b42e1953e42f9e0"
Oct 30 23:58:28.785853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.785784    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115499743e8d570c40a55ce0df5bfda246b58bf7fe02690f7b42e1953e42f9e0"} err="failed to get container status \"115499743e8d570c40a55ce0df5bfda246b58bf7fe02690f7b42e1953e42f9e0\": rpc error: code = NotFound desc = could not find container \"115499743e8d570c40a55ce0df5bfda246b58bf7fe02690f7b42e1953e42f9e0\": container with ID starting with 115499743e8d570c40a55ce0df5bfda246b58bf7fe02690f7b42e1953e42f9e0 not found: ID does not exist"
Oct 30 23:58:28.814010 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.813983    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-credential-socket\") pod \"e88bd702-4a73-4443-9dce-eed8bde3c933\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") "
Oct 30 23:58:28.814177 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.814074    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-workload-certs\") pod \"e88bd702-4a73-4443-9dce-eed8bde3c933\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") "
Oct 30 23:58:28.814177 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.814128    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-workload-socket\") pod \"e88bd702-4a73-4443-9dce-eed8bde3c933\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") "
Oct 30 23:58:28.814177 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.814160    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-podinfo\") pod \"e88bd702-4a73-4443-9dce-eed8bde3c933\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") "
Oct 30 23:58:28.814526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.814191    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-token\" (UniqueName: \"kubernetes.io/projected/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-token\") pod \"e88bd702-4a73-4443-9dce-eed8bde3c933\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") "
Oct 30 23:58:28.814526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.814220    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e88bd702-4a73-4443-9dce-eed8bde3c933-kube-api-access-vvwls\" (UniqueName: \"kubernetes.io/projected/e88bd702-4a73-4443-9dce-eed8bde3c933-kube-api-access-vvwls\") pod \"e88bd702-4a73-4443-9dce-eed8bde3c933\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") "
Oct 30 23:58:28.814526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.814253    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-data\") pod \"e88bd702-4a73-4443-9dce-eed8bde3c933\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") "
Oct 30 23:58:28.814526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.814257    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-credential-socket" pod "e88bd702-4a73-4443-9dce-eed8bde3c933" (UID: "e88bd702-4a73-4443-9dce-eed8bde3c933"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:28.814526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.814301    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e88bd702-4a73-4443-9dce-eed8bde3c933-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88bd702-4a73-4443-9dce-eed8bde3c933-istiod-ca-cert\") pod \"e88bd702-4a73-4443-9dce-eed8bde3c933\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") "
Oct 30 23:58:28.814526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.814332    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-envoy\") pod \"e88bd702-4a73-4443-9dce-eed8bde3c933\" (UID: \"e88bd702-4a73-4443-9dce-eed8bde3c933\") "
Oct 30 23:58:28.814526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.814361    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-workload-certs" pod "e88bd702-4a73-4443-9dce-eed8bde3c933" (UID: "e88bd702-4a73-4443-9dce-eed8bde3c933"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:28.814526 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.814388    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-workload-socket" pod "e88bd702-4a73-4443-9dce-eed8bde3c933" (UID: "e88bd702-4a73-4443-9dce-eed8bde3c933"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:28.814897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.814540    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-data" pod "e88bd702-4a73-4443-9dce-eed8bde3c933" (UID: "e88bd702-4a73-4443-9dce-eed8bde3c933"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:28.814897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.814618    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:28.814897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.814641    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:28.814897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.814657    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:28.814897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.814770    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e88bd702-4a73-4443-9dce-eed8bde3c933-istiod-ca-cert" pod "e88bd702-4a73-4443-9dce-eed8bde3c933" (UID: "e88bd702-4a73-4443-9dce-eed8bde3c933"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:28.816448 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.816416    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-podinfo" pod "e88bd702-4a73-4443-9dce-eed8bde3c933" (UID: "e88bd702-4a73-4443-9dce-eed8bde3c933"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 23:58:28.816670 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.816645    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88bd702-4a73-4443-9dce-eed8bde3c933-kube-api-access-vvwls" pod "e88bd702-4a73-4443-9dce-eed8bde3c933" (UID: "e88bd702-4a73-4443-9dce-eed8bde3c933"). InnerVolumeSpecName "kube-api-access-vvwls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:28.817048 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.817029    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-envoy" pod "e88bd702-4a73-4443-9dce-eed8bde3c933" (UID: "e88bd702-4a73-4443-9dce-eed8bde3c933"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 23:58:28.817104 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.817028    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-token" pod "e88bd702-4a73-4443-9dce-eed8bde3c933" (UID: "e88bd702-4a73-4443-9dce-eed8bde3c933"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:28.915911 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.915876    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88bd702-4a73-4443-9dce-eed8bde3c933-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:28.915911 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.915904    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:28.915911 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.915916    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:28.916158 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.915924    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:28.916158 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.915933    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvwls\" (UniqueName: \"kubernetes.io/projected/e88bd702-4a73-4443-9dce-eed8bde3c933-kube-api-access-vvwls\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:28.916158 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:28.915942    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88bd702-4a73-4443-9dce-eed8bde3c933-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:29.197030 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:29.196995    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"]
Oct 30 23:58:29.201084 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:29.201054    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b81a7a-openshift-default-659d84594c-dsx5p"]
Oct 30 23:58:30.296005 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.295970    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e88bd702-4a73-4443-9dce-eed8bde3c933" path="/var/lib/kubelet/pods/e88bd702-4a73-4443-9dce-eed8bde3c933/volumes"
Oct 30 23:58:30.383185 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.383153    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bbf6fc889-24lsp"]
Oct 30 23:58:30.383444 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.383432    5078 cpu_manager.go:414] "containerMap: removing container" podUID="534c4672-90ed-47ce-9a7b-28aac0ccc886" containerName="revoke-keys"
Oct 30 23:58:30.383489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.383446    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534c4672-90ed-47ce-9a7b-28aac0ccc886" containerName="revoke-keys"
Oct 30 23:58:30.383489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.383467    5078 cpu_manager.go:414] "containerMap: removing container" podUID="dbde5115-111a-42ac-a735-8be11caf2de6" containerName="istio-proxy"
Oct 30 23:58:30.383489 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.383475    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbde5115-111a-42ac-a735-8be11caf2de6" containerName="istio-proxy"
Oct 30 23:58:30.383619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.383496    5078 cpu_manager.go:414] "containerMap: removing container" podUID="e88bd702-4a73-4443-9dce-eed8bde3c933" containerName="istio-proxy"
Oct 30 23:58:30.383619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.383501    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e88bd702-4a73-4443-9dce-eed8bde3c933" containerName="istio-proxy"
Oct 30 23:58:30.383619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.383559    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="e88bd702-4a73-4443-9dce-eed8bde3c933" containerName="istio-proxy"
Oct 30 23:58:30.383619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.383572    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbde5115-111a-42ac-a735-8be11caf2de6" containerName="istio-proxy"
Oct 30 23:58:30.383619 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.383578    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="534c4672-90ed-47ce-9a7b-28aac0ccc886" containerName="revoke-keys"
Oct 30 23:58:30.387568 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.387553    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbf6fc889-24lsp"
Oct 30 23:58:30.469778 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.469753    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 23:58:30.469778 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.469768    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-txfhs\""
Oct 30 23:58:30.489980 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.489954    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 23:58:30.490131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.490035    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 23:58:30.490131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.490069    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 23:58:30.490131 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.490094    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 23:58:30.492026 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.492003    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbf6fc889-24lsp"]
Oct 30 23:58:30.528673 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.528651    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3075ab7a-6f61-4895-b357-95c9fdb22427-metrics-certs\") pod \"maas-controller-7bbf6fc889-24lsp\" (UID: \"3075ab7a-6f61-4895-b357-95c9fdb22427\") " pod="opendatahub/maas-controller-7bbf6fc889-24lsp"
Oct 30 23:58:30.528802 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.528686    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3075ab7a-6f61-4895-b357-95c9fdb22427-webhook-cert\") pod \"maas-controller-7bbf6fc889-24lsp\" (UID: \"3075ab7a-6f61-4895-b357-95c9fdb22427\") " pod="opendatahub/maas-controller-7bbf6fc889-24lsp"
Oct 30 23:58:30.528802 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.528710    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlxt7\" (UniqueName: \"kubernetes.io/projected/3075ab7a-6f61-4895-b357-95c9fdb22427-kube-api-access-xlxt7\") pod \"maas-controller-7bbf6fc889-24lsp\" (UID: \"3075ab7a-6f61-4895-b357-95c9fdb22427\") " pod="opendatahub/maas-controller-7bbf6fc889-24lsp"
Oct 30 23:58:30.630063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.629988    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3075ab7a-6f61-4895-b357-95c9fdb22427-metrics-certs\") pod \"maas-controller-7bbf6fc889-24lsp\" (UID: \"3075ab7a-6f61-4895-b357-95c9fdb22427\") " pod="opendatahub/maas-controller-7bbf6fc889-24lsp"
Oct 30 23:58:30.630063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.630023    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3075ab7a-6f61-4895-b357-95c9fdb22427-webhook-cert\") pod \"maas-controller-7bbf6fc889-24lsp\" (UID: \"3075ab7a-6f61-4895-b357-95c9fdb22427\") " pod="opendatahub/maas-controller-7bbf6fc889-24lsp"
Oct 30 23:58:30.630063 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.630049    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlxt7\" (UniqueName: \"kubernetes.io/projected/3075ab7a-6f61-4895-b357-95c9fdb22427-kube-api-access-xlxt7\") pod \"maas-controller-7bbf6fc889-24lsp\" (UID: \"3075ab7a-6f61-4895-b357-95c9fdb22427\") " pod="opendatahub/maas-controller-7bbf6fc889-24lsp"
Oct 30 23:58:30.632630 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.632586    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3075ab7a-6f61-4895-b357-95c9fdb22427-metrics-certs\") pod \"maas-controller-7bbf6fc889-24lsp\" (UID: \"3075ab7a-6f61-4895-b357-95c9fdb22427\") " pod="opendatahub/maas-controller-7bbf6fc889-24lsp"
Oct 30 23:58:30.632745 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.632712    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3075ab7a-6f61-4895-b357-95c9fdb22427-webhook-cert\") pod \"maas-controller-7bbf6fc889-24lsp\" (UID: \"3075ab7a-6f61-4895-b357-95c9fdb22427\") " pod="opendatahub/maas-controller-7bbf6fc889-24lsp"
Oct 30 23:58:30.688397 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.688369    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlxt7\" (UniqueName: \"kubernetes.io/projected/3075ab7a-6f61-4895-b357-95c9fdb22427-kube-api-access-xlxt7\") pod \"maas-controller-7bbf6fc889-24lsp\" (UID: \"3075ab7a-6f61-4895-b357-95c9fdb22427\") " pod="opendatahub/maas-controller-7bbf6fc889-24lsp"
Oct 30 23:58:30.699402 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:30.699380    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbf6fc889-24lsp"
Oct 30 23:58:31.075935 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:31.075911    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbf6fc889-24lsp"]
Oct 30 23:58:31.078515 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:58:31.078485    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3075ab7a_6f61_4895_b357_95c9fdb22427.slice/crio-25f9505a0be5ee6c39a577cdb03b06a6a04051cf0f25db0db1b2ecf1a6752251 WatchSource:0}: Error finding container 25f9505a0be5ee6c39a577cdb03b06a6a04051cf0f25db0db1b2ecf1a6752251: Status 404 returned error can't find the container with id 25f9505a0be5ee6c39a577cdb03b06a6a04051cf0f25db0db1b2ecf1a6752251
Oct 30 23:58:31.079796 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:31.079778    5078 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 23:58:31.794972 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:31.794936    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbf6fc889-24lsp" event={"ID":"3075ab7a-6f61-4895-b357-95c9fdb22427","Type":"ContainerStarted","Data":"25f9505a0be5ee6c39a577cdb03b06a6a04051cf0f25db0db1b2ecf1a6752251"}
Oct 30 23:58:33.806261 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:33.806228    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbf6fc889-24lsp" event={"ID":"3075ab7a-6f61-4895-b357-95c9fdb22427","Type":"ContainerStarted","Data":"3a8204c0e2f5bcd8e61f1597bfe3dd2864a7585ecc611801972a97274af756fb"}
Oct 30 23:58:33.806761 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:33.806342    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bbf6fc889-24lsp"
Oct 30 23:58:33.924238 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:33.924174    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7bbf6fc889-24lsp" podStartSLOduration=1.8181429630000001 podStartE2EDuration="3.924151964s" totalImagesPullingTime="2.106009001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:33.922627916 +0000 UTC m=+3099.810534266" watchObservedRunningTime="2026-10-30 23:58:33.924151964 +0000 UTC m=+3099.812058327"
Oct 30 23:58:44.045350 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:44.045314    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp"]
Oct 30 23:58:44.094831 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:44.094801    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06cc165c-h7gtp"]
Oct 30 23:58:44.296008 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:44.295933    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87df77dd-cc0e-466b-88de-3397e0ce2fec" path="/var/lib/kubelet/pods/87df77dd-cc0e-466b-88de-3397e0ce2fec/volumes"
Oct 30 23:58:44.813073 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:44.813046    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bbf6fc889-24lsp"
Oct 30 23:58:47.590882 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:58:47.590850    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:58:54.109061 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.109027    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc"]
Oct 30 23:58:54.109561 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.109315    5078 cpu_manager.go:414] "containerMap: removing container" podUID="dbde5115-111a-42ac-a735-8be11caf2de6" containerName="istio-proxy"
Oct 30 23:58:54.109561 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.109325    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbde5115-111a-42ac-a735-8be11caf2de6" containerName="istio-proxy"
Oct 30 23:58:54.109561 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.109384    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbde5115-111a-42ac-a735-8be11caf2de6" containerName="istio-proxy"
Oct 30 23:58:54.112313 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.112297    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc"
Oct 30 23:58:54.206457 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.206402    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnk47\" (UniqueName: \"kubernetes.io/projected/ce3998c5-3854-4ef9-805f-41dace394532-kube-api-access-fnk47\") pod \"maas-api-revoke-sub-85237e45-lcffc\" (UID: \"ce3998c5-3854-4ef9-805f-41dace394532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc"
Oct 30 23:58:54.206671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.206512    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3998c5-3854-4ef9-805f-41dace394532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-lcffc\" (UID: \"ce3998c5-3854-4ef9-805f-41dace394532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc"
Oct 30 23:58:54.206671 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.206587    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3998c5-3854-4ef9-805f-41dace394532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-lcffc\" (UID: \"ce3998c5-3854-4ef9-805f-41dace394532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc"
Oct 30 23:58:54.214111 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.214087    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:58:54.214238 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.214122    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:58:54.214998 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.214981    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:58:54.215082 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.215042    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:58:54.215921 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.215904    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc"]
Oct 30 23:58:54.217222 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.217204    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"]
Oct 30 23:58:54.220889 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.220870    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.265906 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.265878    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"]
Oct 30 23:58:54.269703 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.269682    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.273454 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.273437    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-l8k5z\""
Oct 30 23:58:54.285831 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.285801    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"]
Oct 30 23:58:54.307493 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.307465    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.307663 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.307498    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf5rh\" (UniqueName: \"kubernetes.io/projected/9a60053e-3454-4ef1-9b40-9194f19b840d-kube-api-access-gf5rh\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.307663 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.307537    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.307663 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.307590    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnk47\" (UniqueName: \"kubernetes.io/projected/ce3998c5-3854-4ef9-805f-41dace394532-kube-api-access-fnk47\") pod \"maas-api-revoke-sub-85237e45-lcffc\" (UID: \"ce3998c5-3854-4ef9-805f-41dace394532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc"
Oct 30 23:58:54.307817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.307708    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d25xv\" (UniqueName: \"kubernetes.io/projected/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-kube-api-access-d25xv\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.307817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.307743    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a60053e-3454-4ef1-9b40-9194f19b840d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.307817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.307762    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.307817 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.307781    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a60053e-3454-4ef1-9b40-9194f19b840d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.308015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.307833    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a60053e-3454-4ef1-9b40-9194f19b840d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.308015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.307908    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3998c5-3854-4ef9-805f-41dace394532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-lcffc\" (UID: \"ce3998c5-3854-4ef9-805f-41dace394532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc"
Oct 30 23:58:54.308015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.307929    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a60053e-3454-4ef1-9b40-9194f19b840d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.308015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.307946    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.308015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.307973    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3998c5-3854-4ef9-805f-41dace394532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-lcffc\" (UID: \"ce3998c5-3854-4ef9-805f-41dace394532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc"
Oct 30 23:58:54.308015 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.307989    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.308303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.308060    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.308303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.308132    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a60053e-3454-4ef1-9b40-9194f19b840d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.308303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.308162    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a60053e-3454-4ef1-9b40-9194f19b840d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.308303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.308198    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.308303 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.308257    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a60053e-3454-4ef1-9b40-9194f19b840d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.308531 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.308306    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.308531 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.308352    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a60053e-3454-4ef1-9b40-9194f19b840d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.317801 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.317778    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"]
Oct 30 23:58:54.365012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.364912    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 23:58:54.365012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.364951    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 23:58:54.369041 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.369017    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3998c5-3854-4ef9-805f-41dace394532-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-lcffc\" (UID: \"ce3998c5-3854-4ef9-805f-41dace394532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc"
Oct 30 23:58:54.370754 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.370734    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3998c5-3854-4ef9-805f-41dace394532-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-lcffc\" (UID: \"ce3998c5-3854-4ef9-805f-41dace394532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc"
Oct 30 23:58:54.408738 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.408712    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.408897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.408745    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d25xv\" (UniqueName: \"kubernetes.io/projected/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-kube-api-access-d25xv\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.408897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.408765    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a60053e-3454-4ef1-9b40-9194f19b840d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.408897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.408780    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.408897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.408799    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a60053e-3454-4ef1-9b40-9194f19b840d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.408897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.408823    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a60053e-3454-4ef1-9b40-9194f19b840d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.408897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.408860    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a60053e-3454-4ef1-9b40-9194f19b840d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.408897 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.408883    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.409349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.408919    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.409349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.408945    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.409349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.408978    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a60053e-3454-4ef1-9b40-9194f19b840d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.409349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.409007    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a60053e-3454-4ef1-9b40-9194f19b840d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.409349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.409040    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.409349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.409066    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a60053e-3454-4ef1-9b40-9194f19b840d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.409349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.409106    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.409349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.409132    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a60053e-3454-4ef1-9b40-9194f19b840d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.409349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.409160    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.409349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.409179    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a60053e-3454-4ef1-9b40-9194f19b840d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.409349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.409191    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf5rh\" (UniqueName: \"kubernetes.io/projected/9a60053e-3454-4ef1-9b40-9194f19b840d-kube-api-access-gf5rh\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.409349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.409197    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.409349 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.409266    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a60053e-3454-4ef1-9b40-9194f19b840d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.410117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.409452    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a60053e-3454-4ef1-9b40-9194f19b840d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.410117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.409510    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.410117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.409628    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.410117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.409722    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a60053e-3454-4ef1-9b40-9194f19b840d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.410117 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.410090    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.410408 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.410194    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.410408 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.410343    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a60053e-3454-4ef1-9b40-9194f19b840d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.411481 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.411454    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.411623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.411534    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a60053e-3454-4ef1-9b40-9194f19b840d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.412381 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.412358    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.412529 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.412508    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a60053e-3454-4ef1-9b40-9194f19b840d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.419950 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.419931    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 23:58:54.431314 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.431291    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnk47\" (UniqueName: \"kubernetes.io/projected/ce3998c5-3854-4ef9-805f-41dace394532-kube-api-access-fnk47\") pod \"maas-api-revoke-sub-85237e45-lcffc\" (UID: \"ce3998c5-3854-4ef9-805f-41dace394532\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc"
Oct 30 23:58:54.466868 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.466823    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf5rh\" (UniqueName: \"kubernetes.io/projected/9a60053e-3454-4ef1-9b40-9194f19b840d-kube-api-access-gf5rh\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.467220 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.467141    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.467220 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.467141    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a60053e-3454-4ef1-9b40-9194f19b840d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4662g\" (UID: \"9a60053e-3454-4ef1-9b40-9194f19b840d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.467370 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.467347    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d25xv\" (UniqueName: \"kubernetes.io/projected/c76a2fb3-5cc9-4e1d-bc93-c8acf8723819-kube-api-access-d25xv\") pod \"maas-default-gateway-openshift-default-5df48fcc99-42wvb\" (UID: \"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.584001 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.583976    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-l8k5z\""
Oct 30 23:58:54.586099 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.586077    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:54.593529 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.593511    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:54.775379 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.775355    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 30 23:58:54.784414 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:54.784392    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc"
Oct 30 23:58:55.016575 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:55.016540    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"]
Oct 30 23:58:55.019008 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:58:55.018980    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76a2fb3_5cc9_4e1d_bc93_c8acf8723819.slice/crio-998a7a05b8011b5971e4af4c9ace3eabef09278bc5fd147ce300249c93b6e1f5 WatchSource:0}: Error finding container 998a7a05b8011b5971e4af4c9ace3eabef09278bc5fd147ce300249c93b6e1f5: Status 404 returned error can't find the container with id 998a7a05b8011b5971e4af4c9ace3eabef09278bc5fd147ce300249c93b6e1f5
Oct 30 23:58:55.020653 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:55.020621    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:58:55.020775 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:55.020707    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:58:55.020775 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:55.020751    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:58:55.020894 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:58:55.020814    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a60053e_3454_4ef1_9b40_9194f19b840d.slice/crio-281b82e1de1b33a6af8fff5018f253c65a61d1be369df5cabbc54a1a25f5f811 WatchSource:0}: Error finding container 281b82e1de1b33a6af8fff5018f253c65a61d1be369df5cabbc54a1a25f5f811: Status 404 returned error can't find the container with id 281b82e1de1b33a6af8fff5018f253c65a61d1be369df5cabbc54a1a25f5f811
Oct 30 23:58:55.020963 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:55.020931    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"]
Oct 30 23:58:55.022910 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:55.022804    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:58:55.022910 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:55.022853    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:58:55.022910 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:55.022883    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 30 23:58:55.154147 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:55.154118    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc"]
Oct 30 23:58:55.156258 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:58:55.156223    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3998c5_3854_4ef9_805f_41dace394532.slice/crio-ba1940ab6184ac095ca71ae5afb53cbe31f9b2a24feceac7bab71c426486f5aa WatchSource:0}: Error finding container ba1940ab6184ac095ca71ae5afb53cbe31f9b2a24feceac7bab71c426486f5aa: Status 404 returned error can't find the container with id ba1940ab6184ac095ca71ae5afb53cbe31f9b2a24feceac7bab71c426486f5aa
Oct 30 23:58:55.916088 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:55.915576    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g" event={"ID":"9a60053e-3454-4ef1-9b40-9194f19b840d","Type":"ContainerStarted","Data":"cb127d045a66ca21eee6e0b0b4e28ff30c3fff315d829c35dd2d70d4ae612179"}
Oct 30 23:58:55.916088 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:55.915667    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g" event={"ID":"9a60053e-3454-4ef1-9b40-9194f19b840d","Type":"ContainerStarted","Data":"281b82e1de1b33a6af8fff5018f253c65a61d1be369df5cabbc54a1a25f5f811"}
Oct 30 23:58:55.920750 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:55.920725    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc" event={"ID":"ce3998c5-3854-4ef9-805f-41dace394532","Type":"ContainerStarted","Data":"6dce4d7891d94716744db6a2dee30f5b6671a1b377943c4c4c069ad06e6f5246"}
Oct 30 23:58:55.920887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:55.920766    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc" event={"ID":"ce3998c5-3854-4ef9-805f-41dace394532","Type":"ContainerStarted","Data":"ba1940ab6184ac095ca71ae5afb53cbe31f9b2a24feceac7bab71c426486f5aa"}
Oct 30 23:58:55.922989 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:55.922962    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb" event={"ID":"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819","Type":"ContainerStarted","Data":"2e11102c8e4b3b337d2e1c00c9d700ac848a53023c5f717e5c4fee0f020a264b"}
Oct 30 23:58:55.923102 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:55.923009    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb" event={"ID":"c76a2fb3-5cc9-4e1d-bc93-c8acf8723819","Type":"ContainerStarted","Data":"998a7a05b8011b5971e4af4c9ace3eabef09278bc5fd147ce300249c93b6e1f5"}
Oct 30 23:58:56.047165 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:56.047105    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g" podStartSLOduration=2.047091361 podStartE2EDuration="2.047091361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:56.045364798 +0000 UTC m=+3121.933271155" watchObservedRunningTime="2026-10-30 23:58:56.047091361 +0000 UTC m=+3121.934997711"
Oct 30 23:58:56.300807 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:56.300742    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc" podStartSLOduration=2.300719343 podStartE2EDuration="2.300719343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:56.175826084 +0000 UTC m=+3122.063732439" watchObservedRunningTime="2026-10-30 23:58:56.300719343 +0000 UTC m=+3122.188625698"
Oct 30 23:58:56.593624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:56.593354    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:58:56.593624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:56.593478    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4662g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 23:58:56.593624 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:56.593521    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g" podUID="9a60053e-3454-4ef1-9b40-9194f19b840d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 23:58:56.597621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:56.594673    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:58:56.601627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:56.598690    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-42wvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 23:58:56.601627 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:56.598737    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb" podUID="c76a2fb3-5cc9-4e1d-bc93-c8acf8723819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 23:58:56.930312 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:56.930234    5078 generic.go:356] "Generic (PLEG): container finished" podID="ce3998c5-3854-4ef9-805f-41dace394532" containerID="6dce4d7891d94716744db6a2dee30f5b6671a1b377943c4c4c069ad06e6f5246" exitCode=0
Oct 30 23:58:56.930466 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:56.930322    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc" event={"ID":"ce3998c5-3854-4ef9-805f-41dace394532","Type":"ContainerDied","Data":"6dce4d7891d94716744db6a2dee30f5b6671a1b377943c4c4c069ad06e6f5246"}
Oct 30 23:58:57.047853 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:57.047802    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb" podStartSLOduration=3.04778804 podStartE2EDuration="3.04778804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:56.303766777 +0000 UTC m=+3122.191673134" watchObservedRunningTime="2026-10-30 23:58:57.04778804 +0000 UTC m=+3122.935694392"
Oct 30 23:58:57.587622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:57.586944    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4662g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 23:58:57.587622 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:57.587006    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g" podUID="9a60053e-3454-4ef1-9b40-9194f19b840d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 23:58:57.596985 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:57.596948    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-42wvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 23:58:57.597259 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:57.597232    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb" podUID="c76a2fb3-5cc9-4e1d-bc93-c8acf8723819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 23:58:58.111380 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.111305    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf"]
Oct 30 23:58:58.114450 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.114390    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4a4e7f9-rxbkf"]
Oct 30 23:58:58.155066 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.155044    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc"
Oct 30 23:58:58.243763 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.243685    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce3998c5-3854-4ef9-805f-41dace394532-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3998c5-3854-4ef9-805f-41dace394532-maas-api-xks-ca\") pod \"ce3998c5-3854-4ef9-805f-41dace394532\" (UID: \"ce3998c5-3854-4ef9-805f-41dace394532\") "
Oct 30 23:58:58.243911 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.243766    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3998c5-3854-4ef9-805f-41dace394532-kube-api-access-fnk47\" (UniqueName: \"kubernetes.io/projected/ce3998c5-3854-4ef9-805f-41dace394532-kube-api-access-fnk47\") pod \"ce3998c5-3854-4ef9-805f-41dace394532\" (UID: \"ce3998c5-3854-4ef9-805f-41dace394532\") "
Oct 30 23:58:58.243911 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.243797    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce3998c5-3854-4ef9-805f-41dace394532-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3998c5-3854-4ef9-805f-41dace394532-maas-api-openshift-ca\") pod \"ce3998c5-3854-4ef9-805f-41dace394532\" (UID: \"ce3998c5-3854-4ef9-805f-41dace394532\") "
Oct 30 23:58:58.244242 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.244185    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3998c5-3854-4ef9-805f-41dace394532-maas-api-openshift-ca" pod "ce3998c5-3854-4ef9-805f-41dace394532" (UID: "ce3998c5-3854-4ef9-805f-41dace394532"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 23:58:58.245908 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.245885    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce3998c5-3854-4ef9-805f-41dace394532-maas-api-xks-ca" pod "ce3998c5-3854-4ef9-805f-41dace394532" (UID: "ce3998c5-3854-4ef9-805f-41dace394532"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:58:58.246073 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.246044    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3998c5-3854-4ef9-805f-41dace394532-kube-api-access-fnk47" pod "ce3998c5-3854-4ef9-805f-41dace394532" (UID: "ce3998c5-3854-4ef9-805f-41dace394532"). InnerVolumeSpecName "kube-api-access-fnk47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:58:58.315179 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.315142    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ad7e150-1c8e-49fb-b952-260caedc2274" path="/var/lib/kubelet/pods/7ad7e150-1c8e-49fb-b952-260caedc2274/volumes"
Oct 30 23:58:58.344693 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.344640    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnk47\" (UniqueName: \"kubernetes.io/projected/ce3998c5-3854-4ef9-805f-41dace394532-kube-api-access-fnk47\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:58.344693 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.344664    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce3998c5-3854-4ef9-805f-41dace394532-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:58.344693 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.344675    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce3998c5-3854-4ef9-805f-41dace394532-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:58:58.594637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.589981    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4662g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 23:58:58.594637 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.590050    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g" podUID="9a60053e-3454-4ef1-9b40-9194f19b840d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 23:58:58.595257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.594898    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-42wvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 23:58:58.595257 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.594945    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb" podUID="c76a2fb3-5cc9-4e1d-bc93-c8acf8723819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 23:58:58.948062 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.947981    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc"
Oct 30 23:58:58.948227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.947980    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc" event={"ID":"ce3998c5-3854-4ef9-805f-41dace394532","Type":"ContainerDied","Data":"ba1940ab6184ac095ca71ae5afb53cbe31f9b2a24feceac7bab71c426486f5aa"}
Oct 30 23:58:58.948227 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:58.948126    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba1940ab6184ac095ca71ae5afb53cbe31f9b2a24feceac7bab71c426486f5aa"
Oct 30 23:58:59.590012 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:59.589975    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4662g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 23:58:59.590366 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:59.590336    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g" podUID="9a60053e-3454-4ef1-9b40-9194f19b840d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 23:58:59.593948 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:59.593920    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-42wvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 23:58:59.594144 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:58:59.594121    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb" podUID="c76a2fb3-5cc9-4e1d-bc93-c8acf8723819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 23:59:00.587621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:00.586981    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4662g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 23:59:00.587621 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:00.587039    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g" podUID="9a60053e-3454-4ef1-9b40-9194f19b840d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 23:59:00.599623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:00.595922    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-42wvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 23:59:00.599623 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:00.595990    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb" podUID="c76a2fb3-5cc9-4e1d-bc93-c8acf8723819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 23:59:01.587352 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:01.587315    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4662g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 23:59:01.587551 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:01.587397    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g" podUID="9a60053e-3454-4ef1-9b40-9194f19b840d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 23:59:01.593875 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:01.593842    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-42wvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused" start-of-body=
Oct 30 23:59:01.594391 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:01.593896    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb" podUID="c76a2fb3-5cc9-4e1d-bc93-c8acf8723819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": dial tcp 10.133.1.1:15021: connect: connection refused"
Oct 30 23:59:02.586678 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:02.586644    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4662g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused" start-of-body=
Oct 30 23:59:02.586925 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:02.586697    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g" podUID="9a60053e-3454-4ef1-9b40-9194f19b840d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": dial tcp 10.133.1.2:15021: connect: connection refused"
Oct 30 23:59:03.594354 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:03.594315    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-42wvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:59:03.595009 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:03.594967    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb" podUID="c76a2fb3-5cc9-4e1d-bc93-c8acf8723819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:59:04.633764 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:04.633719    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4662g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:59:04.634211 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:04.633783    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g" podUID="9a60053e-3454-4ef1-9b40-9194f19b840d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:59:04.635734 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:04.635703    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-42wvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:59:04.635887 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:04.635743    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb" podUID="c76a2fb3-5cc9-4e1d-bc93-c8acf8723819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:59:05.064484 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:05.064449    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn"]
Oct 30 23:59:05.067377 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:05.067349    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf2bc85f-dhmzn"]
Oct 30 23:59:05.637157 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:05.637095    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-42wvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:59:05.637722 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:05.637211    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb" podUID="c76a2fb3-5cc9-4e1d-bc93-c8acf8723819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:59:05.675775 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:05.675733    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4662g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 23:59:05.675960 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:05.675818    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g" podUID="9a60053e-3454-4ef1-9b40-9194f19b840d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 23:59:05.683212 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:05.682694    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:59:05.683212 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:05.683146    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:59:05.684095 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:05.684075    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb"
Oct 30 23:59:05.699455 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:05.699433    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:59:05.699823 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:05.699802    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:59:05.710456 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:05.710436    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g"
Oct 30 23:59:06.044143 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:06.044113    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg"]
Oct 30 23:59:06.046285 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:06.046263    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt"]
Oct 30 23:59:06.050799 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:06.050774    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-223234f3-jdbvg"]
Oct 30 23:59:06.052305 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:06.052282    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2043c798-rv2jt"]
Oct 30 23:59:06.296160 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:06.296085    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="016b08e4-7659-4907-9ade-5828ec41bc51" path="/var/lib/kubelet/pods/016b08e4-7659-4907-9ade-5828ec41bc51/volumes"
Oct 30 23:59:06.296452 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:06.296439    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a6bf5bf-83fc-4be2-b5d0-a42a6f821132" path="/var/lib/kubelet/pods/1a6bf5bf-83fc-4be2-b5d0-a42a6f821132/volumes"
Oct 30 23:59:06.296818 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:06.296805    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9e4edba-2552-47c2-bf82-69f0f6f2482f" path="/var/lib/kubelet/pods/a9e4edba-2552-47c2-bf82-69f0f6f2482f/volumes"
Oct 30 23:59:06.736294 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:06.736211    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bbf6fc889-24lsp"]
Oct 30 23:59:06.736727 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:06.736417    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7bbf6fc889-24lsp" podUID="3075ab7a-6f61-4895-b357-95c9fdb22427" containerName="manager" containerID="cri-o://3a8204c0e2f5bcd8e61f1597bfe3dd2864a7585ecc611801972a97274af756fb" gracePeriod=30
Oct 30 23:59:06.983280 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:06.983253    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbf6fc889-24lsp"
Oct 30 23:59:06.995518 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:06.995470    5078 generic.go:356] "Generic (PLEG): container finished" podID="3075ab7a-6f61-4895-b357-95c9fdb22427" containerID="3a8204c0e2f5bcd8e61f1597bfe3dd2864a7585ecc611801972a97274af756fb" exitCode=0
Oct 30 23:59:06.995518 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:06.995510    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbf6fc889-24lsp"
Oct 30 23:59:06.995701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:06.995557    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbf6fc889-24lsp" event={"ID":"3075ab7a-6f61-4895-b357-95c9fdb22427","Type":"ContainerDied","Data":"3a8204c0e2f5bcd8e61f1597bfe3dd2864a7585ecc611801972a97274af756fb"}
Oct 30 23:59:06.995701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:06.995627    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbf6fc889-24lsp" event={"ID":"3075ab7a-6f61-4895-b357-95c9fdb22427","Type":"ContainerDied","Data":"25f9505a0be5ee6c39a577cdb03b06a6a04051cf0f25db0db1b2ecf1a6752251"}
Oct 30 23:59:06.995701 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:06.995677    5078 scope.go:122] "RemoveContainer" containerID="3a8204c0e2f5bcd8e61f1597bfe3dd2864a7585ecc611801972a97274af756fb"
Oct 30 23:59:07.004025 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:07.004007    5078 scope.go:122] "RemoveContainer" containerID="3a8204c0e2f5bcd8e61f1597bfe3dd2864a7585ecc611801972a97274af756fb"
Oct 30 23:59:07.004284 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:59:07.004267    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a8204c0e2f5bcd8e61f1597bfe3dd2864a7585ecc611801972a97274af756fb\": container with ID starting with 3a8204c0e2f5bcd8e61f1597bfe3dd2864a7585ecc611801972a97274af756fb not found: ID does not exist" containerID="3a8204c0e2f5bcd8e61f1597bfe3dd2864a7585ecc611801972a97274af756fb"
Oct 30 23:59:07.004351 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:07.004297    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a8204c0e2f5bcd8e61f1597bfe3dd2864a7585ecc611801972a97274af756fb"} err="failed to get container status \"3a8204c0e2f5bcd8e61f1597bfe3dd2864a7585ecc611801972a97274af756fb\": rpc error: code = NotFound desc = could not find container \"3a8204c0e2f5bcd8e61f1597bfe3dd2864a7585ecc611801972a97274af756fb\": container with ID starting with 3a8204c0e2f5bcd8e61f1597bfe3dd2864a7585ecc611801972a97274af756fb not found: ID does not exist"
Oct 30 23:59:07.043861 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:07.043835    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3075ab7a-6f61-4895-b357-95c9fdb22427-webhook-cert\" (UniqueName: \"kubernetes.io/secret/3075ab7a-6f61-4895-b357-95c9fdb22427-webhook-cert\") pod \"3075ab7a-6f61-4895-b357-95c9fdb22427\" (UID: \"3075ab7a-6f61-4895-b357-95c9fdb22427\") "
Oct 30 23:59:07.043976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:07.043882    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3075ab7a-6f61-4895-b357-95c9fdb22427-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3075ab7a-6f61-4895-b357-95c9fdb22427-metrics-certs\") pod \"3075ab7a-6f61-4895-b357-95c9fdb22427\" (UID: \"3075ab7a-6f61-4895-b357-95c9fdb22427\") "
Oct 30 23:59:07.043976 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:07.043928    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3075ab7a-6f61-4895-b357-95c9fdb22427-kube-api-access-xlxt7\" (UniqueName: \"kubernetes.io/projected/3075ab7a-6f61-4895-b357-95c9fdb22427-kube-api-access-xlxt7\") pod \"3075ab7a-6f61-4895-b357-95c9fdb22427\" (UID: \"3075ab7a-6f61-4895-b357-95c9fdb22427\") "
Oct 30 23:59:07.046207 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:07.046180    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3075ab7a-6f61-4895-b357-95c9fdb22427-kube-api-access-xlxt7" pod "3075ab7a-6f61-4895-b357-95c9fdb22427" (UID: "3075ab7a-6f61-4895-b357-95c9fdb22427"). InnerVolumeSpecName "kube-api-access-xlxt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 23:59:07.046307 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:07.046261    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3075ab7a-6f61-4895-b357-95c9fdb22427-webhook-cert" pod "3075ab7a-6f61-4895-b357-95c9fdb22427" (UID: "3075ab7a-6f61-4895-b357-95c9fdb22427"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:07.046307 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:07.046287    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3075ab7a-6f61-4895-b357-95c9fdb22427-metrics-certs" pod "3075ab7a-6f61-4895-b357-95c9fdb22427" (UID: "3075ab7a-6f61-4895-b357-95c9fdb22427"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 23:59:07.144997 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:07.144957    5078 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3075ab7a-6f61-4895-b357-95c9fdb22427-webhook-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:59:07.144997 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:07.144988    5078 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3075ab7a-6f61-4895-b357-95c9fdb22427-metrics-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:59:07.144997 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:07.144998    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlxt7\" (UniqueName: \"kubernetes.io/projected/3075ab7a-6f61-4895-b357-95c9fdb22427-kube-api-access-xlxt7\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 30 23:59:07.415494 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:07.415459    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7bbf6fc889-24lsp"]
Oct 30 23:59:07.419404 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:07.419380    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7bbf6fc889-24lsp"]
Oct 30 23:59:08.296152 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:08.296121    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3075ab7a-6f61-4895-b357-95c9fdb22427" path="/var/lib/kubelet/pods/3075ab7a-6f61-4895-b357-95c9fdb22427/volumes"
Oct 30 23:59:08.912543 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:08.912510    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7bbf6fc889-972gn"]
Oct 30 23:59:08.912825 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:08.912812    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ce3998c5-3854-4ef9-805f-41dace394532" containerName="revoke-keys"
Oct 30 23:59:08.912892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:08.912827    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3998c5-3854-4ef9-805f-41dace394532" containerName="revoke-keys"
Oct 30 23:59:08.912892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:08.912856    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3075ab7a-6f61-4895-b357-95c9fdb22427" containerName="manager"
Oct 30 23:59:08.912892 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:08.912861    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3075ab7a-6f61-4895-b357-95c9fdb22427" containerName="manager"
Oct 30 23:59:08.913002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:08.912907    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3075ab7a-6f61-4895-b357-95c9fdb22427" containerName="manager"
Oct 30 23:59:08.913002 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:08.912918    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce3998c5-3854-4ef9-805f-41dace394532" containerName="revoke-keys"
Oct 30 23:59:08.917638 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:08.917624    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbf6fc889-972gn"
Oct 30 23:59:08.970590 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:08.970557    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 23:59:08.970590 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:08.970568    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 23:59:09.020327 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:09.020300    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 30 23:59:09.020549 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:09.020525    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 30 23:59:09.020670 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:09.020648    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 30 23:59:09.020783 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:09.020719    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-txfhs\""
Oct 30 23:59:09.022112 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:09.022090    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbf6fc889-972gn"]
Oct 30 23:59:09.060199 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:09.060169    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a43eb438-d90f-4886-bba3-c2b4531028cf-webhook-cert\") pod \"maas-controller-7bbf6fc889-972gn\" (UID: \"a43eb438-d90f-4886-bba3-c2b4531028cf\") " pod="opendatahub/maas-controller-7bbf6fc889-972gn"
Oct 30 23:59:09.060323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:09.060287    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghb6z\" (UniqueName: \"kubernetes.io/projected/a43eb438-d90f-4886-bba3-c2b4531028cf-kube-api-access-ghb6z\") pod \"maas-controller-7bbf6fc889-972gn\" (UID: \"a43eb438-d90f-4886-bba3-c2b4531028cf\") " pod="opendatahub/maas-controller-7bbf6fc889-972gn"
Oct 30 23:59:09.060378 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:09.060330    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a43eb438-d90f-4886-bba3-c2b4531028cf-metrics-certs\") pod \"maas-controller-7bbf6fc889-972gn\" (UID: \"a43eb438-d90f-4886-bba3-c2b4531028cf\") " pod="opendatahub/maas-controller-7bbf6fc889-972gn"
Oct 30 23:59:09.161609 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:09.161571    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghb6z\" (UniqueName: \"kubernetes.io/projected/a43eb438-d90f-4886-bba3-c2b4531028cf-kube-api-access-ghb6z\") pod \"maas-controller-7bbf6fc889-972gn\" (UID: \"a43eb438-d90f-4886-bba3-c2b4531028cf\") " pod="opendatahub/maas-controller-7bbf6fc889-972gn"
Oct 30 23:59:09.161724 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:09.161627    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a43eb438-d90f-4886-bba3-c2b4531028cf-metrics-certs\") pod \"maas-controller-7bbf6fc889-972gn\" (UID: \"a43eb438-d90f-4886-bba3-c2b4531028cf\") " pod="opendatahub/maas-controller-7bbf6fc889-972gn"
Oct 30 23:59:09.161724 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:09.161669    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a43eb438-d90f-4886-bba3-c2b4531028cf-webhook-cert\") pod \"maas-controller-7bbf6fc889-972gn\" (UID: \"a43eb438-d90f-4886-bba3-c2b4531028cf\") " pod="opendatahub/maas-controller-7bbf6fc889-972gn"
Oct 30 23:59:09.164417 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:09.164373    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a43eb438-d90f-4886-bba3-c2b4531028cf-webhook-cert\") pod \"maas-controller-7bbf6fc889-972gn\" (UID: \"a43eb438-d90f-4886-bba3-c2b4531028cf\") " pod="opendatahub/maas-controller-7bbf6fc889-972gn"
Oct 30 23:59:09.164477 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:09.164450    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a43eb438-d90f-4886-bba3-c2b4531028cf-metrics-certs\") pod \"maas-controller-7bbf6fc889-972gn\" (UID: \"a43eb438-d90f-4886-bba3-c2b4531028cf\") " pod="opendatahub/maas-controller-7bbf6fc889-972gn"
Oct 30 23:59:09.218885 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:09.218849    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghb6z\" (UniqueName: \"kubernetes.io/projected/a43eb438-d90f-4886-bba3-c2b4531028cf-kube-api-access-ghb6z\") pod \"maas-controller-7bbf6fc889-972gn\" (UID: \"a43eb438-d90f-4886-bba3-c2b4531028cf\") " pod="opendatahub/maas-controller-7bbf6fc889-972gn"
Oct 30 23:59:09.233747 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:09.233717    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7bbf6fc889-972gn"
Oct 30 23:59:09.605497 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:09.605470    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7bbf6fc889-972gn"]
Oct 30 23:59:09.607865 ip-10-0-130-205 kubenswrapper[5078]: W1030 23:59:09.607839    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43eb438_d90f_4886_bba3_c2b4531028cf.slice/crio-690febb4706234f00b2e5212b0f8bddb4b9e9527177d464d1f3c01ac8a8bc94f WatchSource:0}: Error finding container 690febb4706234f00b2e5212b0f8bddb4b9e9527177d464d1f3c01ac8a8bc94f: Status 404 returned error can't find the container with id 690febb4706234f00b2e5212b0f8bddb4b9e9527177d464d1f3c01ac8a8bc94f
Oct 30 23:59:10.012372 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:10.012289    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbf6fc889-972gn" event={"ID":"a43eb438-d90f-4886-bba3-c2b4531028cf","Type":"ContainerStarted","Data":"690febb4706234f00b2e5212b0f8bddb4b9e9527177d464d1f3c01ac8a8bc94f"}
Oct 30 23:59:11.018695 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:11.018660    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7bbf6fc889-972gn" event={"ID":"a43eb438-d90f-4886-bba3-c2b4531028cf","Type":"ContainerStarted","Data":"ef188d6fb7dc1b673c7b84c6d882b225c22d044ac02325f6b2d9caca68db057e"}
Oct 30 23:59:11.019092 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:11.018781    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7bbf6fc889-972gn"
Oct 30 23:59:11.136856 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:11.136805    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7bbf6fc889-972gn" podStartSLOduration=2.322055632 podStartE2EDuration="3.136789263s" totalImagesPullingTime="814.733631ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:11.135311192 +0000 UTC m=+3137.023217546" watchObservedRunningTime="2026-10-30 23:59:11.136789263 +0000 UTC m=+3137.024695609"
Oct 30 23:59:17.590399 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:59:17.590363    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:59:22.026255 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:22.026218    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7bbf6fc889-972gn"
Oct 30 23:59:47.045323 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:47.045224    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq"]
Oct 30 23:59:47.096365 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:47.096330    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed97f4b-7zsxq"]
Oct 30 23:59:47.590261 ip-10-0-130-205 kubenswrapper[5078]: E1030 23:59:47.590223    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 30 23:59:48.296517 ip-10-0-130-205 kubenswrapper[5078]: I1030 23:59:48.296488    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f751e5aa-c6df-493f-9ecd-0756d593c0ef" path="/var/lib/kubelet/pods/f751e5aa-c6df-493f-9ecd-0756d593c0ef/volumes"
Oct 31 00:00:00.163014 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.162979    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29890080-xmn2r"]
Oct 31 00:00:00.166230 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.166212    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-xmn2r"
Oct 31 00:00:00.223130 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.223102    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\""
Oct 31 00:00:00.268803 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.268776    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-8v5pg\""
Oct 31 00:00:00.270351 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.270324    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29890080-xmn2r"]
Oct 31 00:00:00.271575 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.271549    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4"]
Oct 31 00:00:00.274804 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.274785    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4"
Oct 31 00:00:00.313826 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.313787    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d40082c-dccc-416d-b9d6-5d3a3ca47869-serviceca\") pod \"image-pruner-29890080-xmn2r\" (UID: \"1d40082c-dccc-416d-b9d6-5d3a3ca47869\") " pod="openshift-image-registry/image-pruner-29890080-xmn2r"
Oct 31 00:00:00.313826 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.313826    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2lh8\" (UniqueName: \"kubernetes.io/projected/1d40082c-dccc-416d-b9d6-5d3a3ca47869-kube-api-access-r2lh8\") pod \"image-pruner-29890080-xmn2r\" (UID: \"1d40082c-dccc-416d-b9d6-5d3a3ca47869\") " pod="openshift-image-registry/image-pruner-29890080-xmn2r"
Oct 31 00:00:00.329005 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.328969    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:00:00.329005 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.328978    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:00:00.329005 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.328980    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:00:00.341060 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.341028    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4"]
Oct 31 00:00:00.414434 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.414335    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d40082c-dccc-416d-b9d6-5d3a3ca47869-serviceca\") pod \"image-pruner-29890080-xmn2r\" (UID: \"1d40082c-dccc-416d-b9d6-5d3a3ca47869\") " pod="openshift-image-registry/image-pruner-29890080-xmn2r"
Oct 31 00:00:00.414434 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.414369    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2lh8\" (UniqueName: \"kubernetes.io/projected/1d40082c-dccc-416d-b9d6-5d3a3ca47869-kube-api-access-r2lh8\") pod \"image-pruner-29890080-xmn2r\" (UID: \"1d40082c-dccc-416d-b9d6-5d3a3ca47869\") " pod="openshift-image-registry/image-pruner-29890080-xmn2r"
Oct 31 00:00:00.414434 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.414419    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96715349-8cca-46d3-987b-47bc1cacb416-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890080-zvpb4\" (UID: \"96715349-8cca-46d3-987b-47bc1cacb416\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4"
Oct 31 00:00:00.414773 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.414439    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25rmw\" (UniqueName: \"kubernetes.io/projected/96715349-8cca-46d3-987b-47bc1cacb416-kube-api-access-25rmw\") pod \"maas-api-key-cleanup-29890080-zvpb4\" (UID: \"96715349-8cca-46d3-987b-47bc1cacb416\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4"
Oct 31 00:00:00.414991 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.414971    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d40082c-dccc-416d-b9d6-5d3a3ca47869-serviceca\") pod \"image-pruner-29890080-xmn2r\" (UID: \"1d40082c-dccc-416d-b9d6-5d3a3ca47869\") " pod="openshift-image-registry/image-pruner-29890080-xmn2r"
Oct 31 00:00:00.472921 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.472882    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2lh8\" (UniqueName: \"kubernetes.io/projected/1d40082c-dccc-416d-b9d6-5d3a3ca47869-kube-api-access-r2lh8\") pod \"image-pruner-29890080-xmn2r\" (UID: \"1d40082c-dccc-416d-b9d6-5d3a3ca47869\") " pod="openshift-image-registry/image-pruner-29890080-xmn2r"
Oct 31 00:00:00.506656 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.506613    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-xmn2r"
Oct 31 00:00:00.515099 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.515074    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96715349-8cca-46d3-987b-47bc1cacb416-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890080-zvpb4\" (UID: \"96715349-8cca-46d3-987b-47bc1cacb416\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4"
Oct 31 00:00:00.515224 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.515106    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25rmw\" (UniqueName: \"kubernetes.io/projected/96715349-8cca-46d3-987b-47bc1cacb416-kube-api-access-25rmw\") pod \"maas-api-key-cleanup-29890080-zvpb4\" (UID: \"96715349-8cca-46d3-987b-47bc1cacb416\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4"
Oct 31 00:00:00.515739 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.515712    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96715349-8cca-46d3-987b-47bc1cacb416-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890080-zvpb4\" (UID: \"96715349-8cca-46d3-987b-47bc1cacb416\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4"
Oct 31 00:00:00.577631 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.576323    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25rmw\" (UniqueName: \"kubernetes.io/projected/96715349-8cca-46d3-987b-47bc1cacb416-kube-api-access-25rmw\") pod \"maas-api-key-cleanup-29890080-zvpb4\" (UID: \"96715349-8cca-46d3-987b-47bc1cacb416\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4"
Oct 31 00:00:00.588637 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.587872    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4"
Oct 31 00:00:00.878745 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.878590    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29890080-xmn2r"]
Oct 31 00:00:00.881481 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:00:00.881452    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d40082c_dccc_416d_b9d6_5d3a3ca47869.slice/crio-f7db8a2ce5d9ccc5a790da6cf48899bf0b094fe32fffd36506b61843f3e28707 WatchSource:0}: Error finding container f7db8a2ce5d9ccc5a790da6cf48899bf0b094fe32fffd36506b61843f3e28707: Status 404 returned error can't find the container with id f7db8a2ce5d9ccc5a790da6cf48899bf0b094fe32fffd36506b61843f3e28707
Oct 31 00:00:00.967894 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:00.966977    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4"]
Oct 31 00:00:00.970612 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:00:00.970566    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96715349_8cca_46d3_987b_47bc1cacb416.slice/crio-be30d5cd92c097c9215e3c68c70e2ff0f182b27ac9fa693cef1d0b36bf98ae37 WatchSource:0}: Error finding container be30d5cd92c097c9215e3c68c70e2ff0f182b27ac9fa693cef1d0b36bf98ae37: Status 404 returned error can't find the container with id be30d5cd92c097c9215e3c68c70e2ff0f182b27ac9fa693cef1d0b36bf98ae37
Oct 31 00:00:01.265906 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:01.265877    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-xmn2r" event={"ID":"1d40082c-dccc-416d-b9d6-5d3a3ca47869","Type":"ContainerStarted","Data":"ef6804fa22ecf14903a061ef50351c164b67576a9ab3a3c46d5bf2ee113dace1"}
Oct 31 00:00:01.266335 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:01.265926    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-xmn2r" event={"ID":"1d40082c-dccc-416d-b9d6-5d3a3ca47869","Type":"ContainerStarted","Data":"f7db8a2ce5d9ccc5a790da6cf48899bf0b094fe32fffd36506b61843f3e28707"}
Oct 31 00:00:01.267661 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:01.267633    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4" event={"ID":"96715349-8cca-46d3-987b-47bc1cacb416","Type":"ContainerStarted","Data":"be30d5cd92c097c9215e3c68c70e2ff0f182b27ac9fa693cef1d0b36bf98ae37"}
Oct 31 00:00:01.382570 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:01.382509    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-pruner-29890080-xmn2r" podStartSLOduration=1.3824907739999999 podStartE2EDuration="1.382490774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:01.381031541 +0000 UTC m=+3187.268937894" watchObservedRunningTime="2026-10-31 00:00:01.382490774 +0000 UTC m=+3187.270397129"
Oct 31 00:00:02.275183 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:02.275150    5078 generic.go:356] "Generic (PLEG): container finished" podID="96715349-8cca-46d3-987b-47bc1cacb416" containerID="c8801a62b17cff44686425bdf49a9ad1eb8ea91056dc8aadda08ee98d527171c" exitCode=60
Oct 31 00:00:02.275726 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:02.275235    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4" event={"ID":"96715349-8cca-46d3-987b-47bc1cacb416","Type":"ContainerDied","Data":"c8801a62b17cff44686425bdf49a9ad1eb8ea91056dc8aadda08ee98d527171c"}
Oct 31 00:00:02.275726 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:02.275449    5078 scope.go:122] "RemoveContainer" containerID="c8801a62b17cff44686425bdf49a9ad1eb8ea91056dc8aadda08ee98d527171c"
Oct 31 00:00:03.281465 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:03.281430    5078 generic.go:356] "Generic (PLEG): container finished" podID="1d40082c-dccc-416d-b9d6-5d3a3ca47869" containerID="ef6804fa22ecf14903a061ef50351c164b67576a9ab3a3c46d5bf2ee113dace1" exitCode=0
Oct 31 00:00:03.281981 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:03.281508    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-xmn2r" event={"ID":"1d40082c-dccc-416d-b9d6-5d3a3ca47869","Type":"ContainerDied","Data":"ef6804fa22ecf14903a061ef50351c164b67576a9ab3a3c46d5bf2ee113dace1"}
Oct 31 00:00:03.283401 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:03.283375    5078 generic.go:356] "Generic (PLEG): container finished" podID="96715349-8cca-46d3-987b-47bc1cacb416" containerID="23b9102a8c8f9d8b8926a96c02d315aed619552e862df24454f885eed63d48d0" exitCode=60
Oct 31 00:00:03.283519 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:03.283446    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4" event={"ID":"96715349-8cca-46d3-987b-47bc1cacb416","Type":"ContainerDied","Data":"23b9102a8c8f9d8b8926a96c02d315aed619552e862df24454f885eed63d48d0"}
Oct 31 00:00:03.283519 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:03.283505    5078 scope.go:122] "RemoveContainer" containerID="c8801a62b17cff44686425bdf49a9ad1eb8ea91056dc8aadda08ee98d527171c"
Oct 31 00:00:03.283740 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:03.283724    5078 scope.go:122] "RemoveContainer" containerID="23b9102a8c8f9d8b8926a96c02d315aed619552e862df24454f885eed63d48d0"
Oct 31 00:00:03.283974 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:00:03.283958    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890080-zvpb4_odh-ai-gateway-infra(96715349-8cca-46d3-987b-47bc1cacb416)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4" podUID="96715349-8cca-46d3-987b-47bc1cacb416"
Oct 31 00:00:03.932086 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:03.932054    5078 scope.go:122] "RemoveContainer" containerID="e58ce3f2c8553035b8ee3d532ad9351a998f25a47cff936a1e81d90356b76702"
Oct 31 00:00:03.940851 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:03.940829    5078 scope.go:122] "RemoveContainer" containerID="23a439e80f03426effbe94af954805c4a6e99c41543236836cfa885c658b2bc3"
Oct 31 00:00:03.949048 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:03.949029    5078 scope.go:122] "RemoveContainer" containerID="60c6f6c530d113a935c5b7cbc14397b46ec352cd661c0e9103b5d747e41ad803"
Oct 31 00:00:03.958747 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:03.958060    5078 scope.go:122] "RemoveContainer" containerID="c421d31b32e3b376d947231c2c8506e5a07ebd5990815903429ced1e27225156"
Oct 31 00:00:03.969468 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:03.969443    5078 scope.go:122] "RemoveContainer" containerID="2c4a245ee96b287f58952f8bb52d2c96307a361e718174d437b8987ae3bf0c2d"
Oct 31 00:00:03.977769 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:03.977739    5078 scope.go:122] "RemoveContainer" containerID="e4c40014ed812177b589863fdc4d892c69ef694c5035aea93d31d8779ba7fb3c"
Oct 31 00:00:04.290821 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:04.290794    5078 scope.go:122] "RemoveContainer" containerID="23b9102a8c8f9d8b8926a96c02d315aed619552e862df24454f885eed63d48d0"
Oct 31 00:00:04.291328 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:00:04.291006    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890080-zvpb4_odh-ai-gateway-infra(96715349-8cca-46d3-987b-47bc1cacb416)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4" podUID="96715349-8cca-46d3-987b-47bc1cacb416"
Oct 31 00:00:04.430292 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:04.430260    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-xmn2r"
Oct 31 00:00:04.555820 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:04.555729    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d40082c-dccc-416d-b9d6-5d3a3ca47869-serviceca\" (UniqueName: \"kubernetes.io/configmap/1d40082c-dccc-416d-b9d6-5d3a3ca47869-serviceca\") pod \"1d40082c-dccc-416d-b9d6-5d3a3ca47869\" (UID: \"1d40082c-dccc-416d-b9d6-5d3a3ca47869\") "
Oct 31 00:00:04.555984 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:04.555848    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d40082c-dccc-416d-b9d6-5d3a3ca47869-kube-api-access-r2lh8\" (UniqueName: \"kubernetes.io/projected/1d40082c-dccc-416d-b9d6-5d3a3ca47869-kube-api-access-r2lh8\") pod \"1d40082c-dccc-416d-b9d6-5d3a3ca47869\" (UID: \"1d40082c-dccc-416d-b9d6-5d3a3ca47869\") "
Oct 31 00:00:04.556160 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:04.556135    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d40082c-dccc-416d-b9d6-5d3a3ca47869-serviceca" pod "1d40082c-dccc-416d-b9d6-5d3a3ca47869" (UID: "1d40082c-dccc-416d-b9d6-5d3a3ca47869"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:00:04.558232 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:04.558205    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d40082c-dccc-416d-b9d6-5d3a3ca47869-kube-api-access-r2lh8" pod "1d40082c-dccc-416d-b9d6-5d3a3ca47869" (UID: "1d40082c-dccc-416d-b9d6-5d3a3ca47869"). InnerVolumeSpecName "kube-api-access-r2lh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:04.657051 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:04.657004    5078 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1d40082c-dccc-416d-b9d6-5d3a3ca47869-serviceca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:00:04.657051 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:04.657045    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2lh8\" (UniqueName: \"kubernetes.io/projected/1d40082c-dccc-416d-b9d6-5d3a3ca47869-kube-api-access-r2lh8\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:00:05.296213 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:05.296175    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29890080-xmn2r" event={"ID":"1d40082c-dccc-416d-b9d6-5d3a3ca47869","Type":"ContainerDied","Data":"f7db8a2ce5d9ccc5a790da6cf48899bf0b094fe32fffd36506b61843f3e28707"}
Oct 31 00:00:05.296213 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:05.296206    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29890080-xmn2r"
Oct 31 00:00:05.296750 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:05.296235    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7db8a2ce5d9ccc5a790da6cf48899bf0b094fe32fffd36506b61843f3e28707"
Oct 31 00:00:14.293932 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:14.293901    5078 scope.go:122] "RemoveContainer" containerID="23b9102a8c8f9d8b8926a96c02d315aed619552e862df24454f885eed63d48d0"
Oct 31 00:00:15.346996 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:15.346960    5078 generic.go:356] "Generic (PLEG): container finished" podID="96715349-8cca-46d3-987b-47bc1cacb416" containerID="6a0f4378a67565a15c0960d4b8b1df617a11b38e0d653829b0cfc9983bfbef84" exitCode=60
Oct 31 00:00:15.347436 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:15.347026    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4" event={"ID":"96715349-8cca-46d3-987b-47bc1cacb416","Type":"ContainerDied","Data":"6a0f4378a67565a15c0960d4b8b1df617a11b38e0d653829b0cfc9983bfbef84"}
Oct 31 00:00:15.347436 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:15.347074    5078 scope.go:122] "RemoveContainer" containerID="23b9102a8c8f9d8b8926a96c02d315aed619552e862df24454f885eed63d48d0"
Oct 31 00:00:15.347436 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:15.347385    5078 scope.go:122] "RemoveContainer" containerID="6a0f4378a67565a15c0960d4b8b1df617a11b38e0d653829b0cfc9983bfbef84"
Oct 31 00:00:15.347655 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:00:15.347634    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890080-zvpb4_odh-ai-gateway-infra(96715349-8cca-46d3-987b-47bc1cacb416)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4" podUID="96715349-8cca-46d3-987b-47bc1cacb416"
Oct 31 00:00:16.473304 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:16.473267    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4"]
Oct 31 00:00:16.605726 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:16.605702    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4"
Oct 31 00:00:16.773926 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:16.773844    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96715349-8cca-46d3-987b-47bc1cacb416-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96715349-8cca-46d3-987b-47bc1cacb416-maas-api-service-ca\") pod \"96715349-8cca-46d3-987b-47bc1cacb416\" (UID: \"96715349-8cca-46d3-987b-47bc1cacb416\") "
Oct 31 00:00:16.774079 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:16.773954    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96715349-8cca-46d3-987b-47bc1cacb416-kube-api-access-25rmw\" (UniqueName: \"kubernetes.io/projected/96715349-8cca-46d3-987b-47bc1cacb416-kube-api-access-25rmw\") pod \"96715349-8cca-46d3-987b-47bc1cacb416\" (UID: \"96715349-8cca-46d3-987b-47bc1cacb416\") "
Oct 31 00:00:16.774217 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:16.774196    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96715349-8cca-46d3-987b-47bc1cacb416-maas-api-service-ca" pod "96715349-8cca-46d3-987b-47bc1cacb416" (UID: "96715349-8cca-46d3-987b-47bc1cacb416"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:00:16.776159 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:16.776134    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96715349-8cca-46d3-987b-47bc1cacb416-kube-api-access-25rmw" pod "96715349-8cca-46d3-987b-47bc1cacb416" (UID: "96715349-8cca-46d3-987b-47bc1cacb416"). InnerVolumeSpecName "kube-api-access-25rmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:16.875500 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:16.875460    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96715349-8cca-46d3-987b-47bc1cacb416-maas-api-service-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:00:16.875500 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:16.875492    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25rmw\" (UniqueName: \"kubernetes.io/projected/96715349-8cca-46d3-987b-47bc1cacb416-kube-api-access-25rmw\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:00:17.360553 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:17.360515    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4" event={"ID":"96715349-8cca-46d3-987b-47bc1cacb416","Type":"ContainerDied","Data":"be30d5cd92c097c9215e3c68c70e2ff0f182b27ac9fa693cef1d0b36bf98ae37"}
Oct 31 00:00:17.360807 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:17.360581    5078 scope.go:122] "RemoveContainer" containerID="6a0f4378a67565a15c0960d4b8b1df617a11b38e0d653829b0cfc9983bfbef84"
Oct 31 00:00:17.360807 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:17.360528    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4"
Oct 31 00:00:17.525910 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:17.525880    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4"]
Oct 31 00:00:17.529895 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:17.529860    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890080-zvpb4"]
Oct 31 00:00:17.590079 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:00:17.590037    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:00:18.296731 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:18.296699    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96715349-8cca-46d3-987b-47bc1cacb416" path="/var/lib/kubelet/pods/96715349-8cca-46d3-987b-47bc1cacb416/volumes"
Oct 31 00:00:34.039982 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.039942    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn"]
Oct 31 00:00:34.040383 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.040240    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1d40082c-dccc-416d-b9d6-5d3a3ca47869" containerName="image-pruner"
Oct 31 00:00:34.040383 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.040251    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d40082c-dccc-416d-b9d6-5d3a3ca47869" containerName="image-pruner"
Oct 31 00:00:34.040383 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.040261    5078 cpu_manager.go:414] "containerMap: removing container" podUID="96715349-8cca-46d3-987b-47bc1cacb416" containerName="cleanup"
Oct 31 00:00:34.040383 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.040267    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96715349-8cca-46d3-987b-47bc1cacb416" containerName="cleanup"
Oct 31 00:00:34.040383 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.040274    5078 cpu_manager.go:414] "containerMap: removing container" podUID="96715349-8cca-46d3-987b-47bc1cacb416" containerName="cleanup"
Oct 31 00:00:34.040383 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.040279    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96715349-8cca-46d3-987b-47bc1cacb416" containerName="cleanup"
Oct 31 00:00:34.040383 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.040347    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="96715349-8cca-46d3-987b-47bc1cacb416" containerName="cleanup"
Oct 31 00:00:34.040383 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.040357    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d40082c-dccc-416d-b9d6-5d3a3ca47869" containerName="image-pruner"
Oct 31 00:00:34.040383 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.040363    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="96715349-8cca-46d3-987b-47bc1cacb416" containerName="cleanup"
Oct 31 00:00:34.040383 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.040368    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="96715349-8cca-46d3-987b-47bc1cacb416" containerName="cleanup"
Oct 31 00:00:34.043428 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.043412    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn"
Oct 31 00:00:34.096879 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.096842    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:00:34.123766 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.123734    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bff7k\" (UniqueName: \"kubernetes.io/projected/75b6a3b8-bd2d-445d-b368-d7c7ef375882-kube-api-access-bff7k\") pod \"maas-api-revoke-sub-2e1f5b5f-psdgn\" (UID: \"75b6a3b8-bd2d-445d-b368-d7c7ef375882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn"
Oct 31 00:00:34.123915 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.123787    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b6a3b8-bd2d-445d-b368-d7c7ef375882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-psdgn\" (UID: \"75b6a3b8-bd2d-445d-b368-d7c7ef375882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn"
Oct 31 00:00:34.123958 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.123925    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b6a3b8-bd2d-445d-b368-d7c7ef375882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-psdgn\" (UID: \"75b6a3b8-bd2d-445d-b368-d7c7ef375882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn"
Oct 31 00:00:34.146000 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.145975    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:00:34.146150 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.145993    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:00:34.146150 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.146010    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:00:34.147843 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.147821    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn"]
Oct 31 00:00:34.225270 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.225235    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b6a3b8-bd2d-445d-b368-d7c7ef375882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-psdgn\" (UID: \"75b6a3b8-bd2d-445d-b368-d7c7ef375882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn"
Oct 31 00:00:34.225270 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.225272    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bff7k\" (UniqueName: \"kubernetes.io/projected/75b6a3b8-bd2d-445d-b368-d7c7ef375882-kube-api-access-bff7k\") pod \"maas-api-revoke-sub-2e1f5b5f-psdgn\" (UID: \"75b6a3b8-bd2d-445d-b368-d7c7ef375882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn"
Oct 31 00:00:34.225470 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.225308    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b6a3b8-bd2d-445d-b368-d7c7ef375882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-psdgn\" (UID: \"75b6a3b8-bd2d-445d-b368-d7c7ef375882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn"
Oct 31 00:00:34.225884 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.225865    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b6a3b8-bd2d-445d-b368-d7c7ef375882-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-psdgn\" (UID: \"75b6a3b8-bd2d-445d-b368-d7c7ef375882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn"
Oct 31 00:00:34.227853 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.227830    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b6a3b8-bd2d-445d-b368-d7c7ef375882-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-psdgn\" (UID: \"75b6a3b8-bd2d-445d-b368-d7c7ef375882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn"
Oct 31 00:00:34.282146 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.282120    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bff7k\" (UniqueName: \"kubernetes.io/projected/75b6a3b8-bd2d-445d-b368-d7c7ef375882-kube-api-access-bff7k\") pod \"maas-api-revoke-sub-2e1f5b5f-psdgn\" (UID: \"75b6a3b8-bd2d-445d-b368-d7c7ef375882\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn"
Oct 31 00:00:34.355411 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.355384    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn"
Oct 31 00:00:34.727922 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:34.727896    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn"]
Oct 31 00:00:34.730253 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:00:34.730225    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b6a3b8_bd2d_445d_b368_d7c7ef375882.slice/crio-48df1eba3112dad91e20b2f76d3de7265d967729bbe8e446deaad0478f524faf WatchSource:0}: Error finding container 48df1eba3112dad91e20b2f76d3de7265d967729bbe8e446deaad0478f524faf: Status 404 returned error can't find the container with id 48df1eba3112dad91e20b2f76d3de7265d967729bbe8e446deaad0478f524faf
Oct 31 00:00:35.449075 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:35.449038    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn" event={"ID":"75b6a3b8-bd2d-445d-b368-d7c7ef375882","Type":"ContainerStarted","Data":"739b588ea746c57ad7292a28dd7e62906f4c2a0ebbd2b14208691b01c2ff92b9"}
Oct 31 00:00:35.449487 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:35.449095    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn" event={"ID":"75b6a3b8-bd2d-445d-b368-d7c7ef375882","Type":"ContainerStarted","Data":"48df1eba3112dad91e20b2f76d3de7265d967729bbe8e446deaad0478f524faf"}
Oct 31 00:00:35.580330 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:35.580279    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn" podStartSLOduration=2.580265931 podStartE2EDuration="2.580265931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:35.577992274 +0000 UTC m=+3221.465898628" watchObservedRunningTime="2026-10-31 00:00:35.580265931 +0000 UTC m=+3221.468172283"
Oct 31 00:00:36.455250 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:36.455216    5078 generic.go:356] "Generic (PLEG): container finished" podID="75b6a3b8-bd2d-445d-b368-d7c7ef375882" containerID="739b588ea746c57ad7292a28dd7e62906f4c2a0ebbd2b14208691b01c2ff92b9" exitCode=0
Oct 31 00:00:36.455678 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:36.455313    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn" event={"ID":"75b6a3b8-bd2d-445d-b368-d7c7ef375882","Type":"ContainerDied","Data":"739b588ea746c57ad7292a28dd7e62906f4c2a0ebbd2b14208691b01c2ff92b9"}
Oct 31 00:00:37.582218 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:37.582194    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn"
Oct 31 00:00:37.655332 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:37.655296    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b6a3b8-bd2d-445d-b368-d7c7ef375882-kube-api-access-bff7k\" (UniqueName: \"kubernetes.io/projected/75b6a3b8-bd2d-445d-b368-d7c7ef375882-kube-api-access-bff7k\") pod \"75b6a3b8-bd2d-445d-b368-d7c7ef375882\" (UID: \"75b6a3b8-bd2d-445d-b368-d7c7ef375882\") "
Oct 31 00:00:37.655525 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:37.655386    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75b6a3b8-bd2d-445d-b368-d7c7ef375882-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b6a3b8-bd2d-445d-b368-d7c7ef375882-maas-api-xks-ca\") pod \"75b6a3b8-bd2d-445d-b368-d7c7ef375882\" (UID: \"75b6a3b8-bd2d-445d-b368-d7c7ef375882\") "
Oct 31 00:00:37.655525 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:37.655490    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b6a3b8-bd2d-445d-b368-d7c7ef375882-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b6a3b8-bd2d-445d-b368-d7c7ef375882-maas-api-openshift-ca\") pod \"75b6a3b8-bd2d-445d-b368-d7c7ef375882\" (UID: \"75b6a3b8-bd2d-445d-b368-d7c7ef375882\") "
Oct 31 00:00:37.655890 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:37.655862    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b6a3b8-bd2d-445d-b368-d7c7ef375882-maas-api-openshift-ca" pod "75b6a3b8-bd2d-445d-b368-d7c7ef375882" (UID: "75b6a3b8-bd2d-445d-b368-d7c7ef375882"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:00:37.657668 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:37.657639    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75b6a3b8-bd2d-445d-b368-d7c7ef375882-maas-api-xks-ca" pod "75b6a3b8-bd2d-445d-b368-d7c7ef375882" (UID: "75b6a3b8-bd2d-445d-b368-d7c7ef375882"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:00:37.657817 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:37.657773    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b6a3b8-bd2d-445d-b368-d7c7ef375882-kube-api-access-bff7k" pod "75b6a3b8-bd2d-445d-b368-d7c7ef375882" (UID: "75b6a3b8-bd2d-445d-b368-d7c7ef375882"). InnerVolumeSpecName "kube-api-access-bff7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:37.756754 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:37.756667    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bff7k\" (UniqueName: \"kubernetes.io/projected/75b6a3b8-bd2d-445d-b368-d7c7ef375882-kube-api-access-bff7k\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:00:37.756754 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:37.756699    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75b6a3b8-bd2d-445d-b368-d7c7ef375882-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:00:37.756754 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:37.756714    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75b6a3b8-bd2d-445d-b368-d7c7ef375882-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:00:38.465667 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:38.465637    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn"
Oct 31 00:00:38.465835 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:38.465637    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn" event={"ID":"75b6a3b8-bd2d-445d-b368-d7c7ef375882","Type":"ContainerDied","Data":"48df1eba3112dad91e20b2f76d3de7265d967729bbe8e446deaad0478f524faf"}
Oct 31 00:00:38.465835 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:38.465769    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48df1eba3112dad91e20b2f76d3de7265d967729bbe8e446deaad0478f524faf"
Oct 31 00:00:39.954983 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:39.954950    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"]
Oct 31 00:00:39.955381 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:39.955248    5078 cpu_manager.go:414] "containerMap: removing container" podUID="75b6a3b8-bd2d-445d-b368-d7c7ef375882" containerName="revoke-keys"
Oct 31 00:00:39.955381 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:39.955258    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b6a3b8-bd2d-445d-b368-d7c7ef375882" containerName="revoke-keys"
Oct 31 00:00:39.955381 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:39.955283    5078 cpu_manager.go:414] "containerMap: removing container" podUID="96715349-8cca-46d3-987b-47bc1cacb416" containerName="cleanup"
Oct 31 00:00:39.955381 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:39.955287    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96715349-8cca-46d3-987b-47bc1cacb416" containerName="cleanup"
Oct 31 00:00:39.955381 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:39.955355    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b6a3b8-bd2d-445d-b368-d7c7ef375882" containerName="revoke-keys"
Oct 31 00:00:39.958094 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:39.958079    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.004859 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.004831    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"]
Oct 31 00:00:40.007938 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.007923    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.061955 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.061924    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"]
Oct 31 00:00:40.075270 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.075243    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af70286-d197-473c-be7d-9333a83d1c4d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.075486 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.075463    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.075700 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.075666    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.075807 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.075702    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af70286-d197-473c-be7d-9333a83d1c4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.075888 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.075817    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af70286-d197-473c-be7d-9333a83d1c4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.075949 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.075896    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.075949 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.075929    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7vc6\" (UniqueName: \"kubernetes.io/projected/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-kube-api-access-w7vc6\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.075949 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.075946    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af70286-d197-473c-be7d-9333a83d1c4d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.076107 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.075965    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.076107 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.075986    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af70286-d197-473c-be7d-9333a83d1c4d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.076107 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.076012    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl99c\" (UniqueName: \"kubernetes.io/projected/9af70286-d197-473c-be7d-9333a83d1c4d-kube-api-access-jl99c\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.076107 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.076034    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af70286-d197-473c-be7d-9333a83d1c4d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.076107 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.076047    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.076107 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.076062    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.076107 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.076088    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.076485 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.076128    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af70286-d197-473c-be7d-9333a83d1c4d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.076485 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.076146    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.076485 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.076168    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af70286-d197-473c-be7d-9333a83d1c4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.115635 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.115574    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"]
Oct 31 00:00:40.176907 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.176874    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.177093 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.176921    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af70286-d197-473c-be7d-9333a83d1c4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.177093 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.176968    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af70286-d197-473c-be7d-9333a83d1c4d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.177093 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.176992    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.177093 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.177028    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.177093 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.177052    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af70286-d197-473c-be7d-9333a83d1c4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.177093 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.177083    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af70286-d197-473c-be7d-9333a83d1c4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.177410 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.177112    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.177410 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.177152    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7vc6\" (UniqueName: \"kubernetes.io/projected/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-kube-api-access-w7vc6\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.177410 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.177179    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af70286-d197-473c-be7d-9333a83d1c4d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.177410 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.177205    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.177410 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.177239    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af70286-d197-473c-be7d-9333a83d1c4d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.177410 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.177270    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl99c\" (UniqueName: \"kubernetes.io/projected/9af70286-d197-473c-be7d-9333a83d1c4d-kube-api-access-jl99c\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.177410 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.177293    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.177410 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.177303    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af70286-d197-473c-be7d-9333a83d1c4d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.177410 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.177332    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.177410 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.177360    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af70286-d197-473c-be7d-9333a83d1c4d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.177410 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.177398    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.178453 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.177465    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.178453 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.177533    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af70286-d197-473c-be7d-9333a83d1c4d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.178453 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.177640    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.178453 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.177851    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.178453 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.178078    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9af70286-d197-473c-be7d-9333a83d1c4d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.178453 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.178308    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9af70286-d197-473c-be7d-9333a83d1c4d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.178453 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.178357    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.179025 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.179002    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.179143 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.179115    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9af70286-d197-473c-be7d-9333a83d1c4d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.179360 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.179339    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9af70286-d197-473c-be7d-9333a83d1c4d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.181112 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.181073    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9af70286-d197-473c-be7d-9333a83d1c4d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.181422 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.181397    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.181662 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.181640    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.182118 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.182078    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9af70286-d197-473c-be7d-9333a83d1c4d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.235668 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.235580    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.235818 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.235692    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl99c\" (UniqueName: \"kubernetes.io/projected/9af70286-d197-473c-be7d-9333a83d1c4d-kube-api-access-jl99c\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.235818 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.235752    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9af70286-d197-473c-be7d-9333a83d1c4d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4g4vj\" (UID: \"9af70286-d197-473c-be7d-9333a83d1c4d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.235818 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.235771    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7vc6\" (UniqueName: \"kubernetes.io/projected/c9e4b867-ea81-4fc3-91c4-ce1129b13d3f-kube-api-access-w7vc6\") pod \"maas-default-gateway-openshift-default-5df48fcc99-659rv\" (UID: \"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.270758 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.270732    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:40.320737 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.320573    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:40.696511 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.696487    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"]
Oct 31 00:00:40.698557 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:00:40.698530    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af70286_d197_473c_be7d_9333a83d1c4d.slice/crio-5e0bd9687677e3bc407e71ee0422a261be75d4bf1e4eda466eaf3415eb7c19b4 WatchSource:0}: Error finding container 5e0bd9687677e3bc407e71ee0422a261be75d4bf1e4eda466eaf3415eb7c19b4: Status 404 returned error can't find the container with id 5e0bd9687677e3bc407e71ee0422a261be75d4bf1e4eda466eaf3415eb7c19b4
Oct 31 00:00:40.700687 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.700653    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:00:40.700811 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.700734    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:00:40.700811 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.700780    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:00:40.749662 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.749632    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"]
Oct 31 00:00:40.751963 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:00:40.751939    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e4b867_ea81_4fc3_91c4_ce1129b13d3f.slice/crio-10666646ded86e3a48c6261a9db6c5ae72fdd71c10bf42ad654f0a70c5478c6d WatchSource:0}: Error finding container 10666646ded86e3a48c6261a9db6c5ae72fdd71c10bf42ad654f0a70c5478c6d: Status 404 returned error can't find the container with id 10666646ded86e3a48c6261a9db6c5ae72fdd71c10bf42ad654f0a70c5478c6d
Oct 31 00:00:40.754244 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.754212    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:00:40.754327 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.754281    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:00:40.754327 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:40.754306    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:00:41.512062 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:41.511984    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv" event={"ID":"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f","Type":"ContainerStarted","Data":"66432afb97f2977ec46995adcb3f79a7635224c37aadef89aa0d4e61143b40a0"}
Oct 31 00:00:41.512062 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:41.512053    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv" event={"ID":"c9e4b867-ea81-4fc3-91c4-ce1129b13d3f","Type":"ContainerStarted","Data":"10666646ded86e3a48c6261a9db6c5ae72fdd71c10bf42ad654f0a70c5478c6d"}
Oct 31 00:00:41.514021 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:41.513993    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj" event={"ID":"9af70286-d197-473c-be7d-9333a83d1c4d","Type":"ContainerStarted","Data":"5e686fb5807f341cd85ba0265cf4b5f7e2a0973dad0a6af952631be0a0daeece"}
Oct 31 00:00:41.514151 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:41.514043    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj" event={"ID":"9af70286-d197-473c-be7d-9333a83d1c4d","Type":"ContainerStarted","Data":"5e0bd9687677e3bc407e71ee0422a261be75d4bf1e4eda466eaf3415eb7c19b4"}
Oct 31 00:00:41.876349 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:41.876287    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv" podStartSLOduration=2.876265908 podStartE2EDuration="2.876265908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:41.74865383 +0000 UTC m=+3227.636560183" watchObservedRunningTime="2026-10-31 00:00:41.876265908 +0000 UTC m=+3227.764172264"
Oct 31 00:00:42.277679 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:42.277469    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:42.277873 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:42.277691    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4g4vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 00:00:42.277873 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:42.277744    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj" podUID="9af70286-d197-473c-be7d-9333a83d1c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 00:00:42.321190 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:42.321155    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:42.322382 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:42.322360    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-659rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 31 00:00:42.322435 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:42.322414    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv" podUID="c9e4b867-ea81-4fc3-91c4-ce1129b13d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 31 00:00:43.274047 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:43.274009    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4g4vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 00:00:43.274701 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:43.274668    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj" podUID="9af70286-d197-473c-be7d-9333a83d1c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 00:00:43.321428 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:43.321385    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-659rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 31 00:00:43.321652 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:43.321469    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv" podUID="c9e4b867-ea81-4fc3-91c4-ce1129b13d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 31 00:00:44.273620 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:44.272997    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4g4vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 00:00:44.273620 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:44.273060    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj" podUID="9af70286-d197-473c-be7d-9333a83d1c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 00:00:44.322014 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:44.321983    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-659rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 31 00:00:44.322387 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:44.322053    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv" podUID="c9e4b867-ea81-4fc3-91c4-ce1129b13d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 31 00:00:45.272071 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:45.272030    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4g4vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 00:00:45.272351 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:45.272323    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj" podUID="9af70286-d197-473c-be7d-9333a83d1c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 00:00:45.321208 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:45.321174    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-659rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 31 00:00:45.321404 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:45.321233    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv" podUID="c9e4b867-ea81-4fc3-91c4-ce1129b13d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 31 00:00:46.274623 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.273995    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4g4vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 00:00:46.274623 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.274057    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj" podUID="9af70286-d197-473c-be7d-9333a83d1c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 00:00:46.321346 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.321306    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-659rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 31 00:00:46.321525 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.321359    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv" podUID="c9e4b867-ea81-4fc3-91c4-ce1129b13d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 31 00:00:46.673714 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.673638    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj" podStartSLOduration=7.673616895 podStartE2EDuration="7.673616895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:00:41.881131754 +0000 UTC m=+3227.769038110" watchObservedRunningTime="2026-10-31 00:00:46.673616895 +0000 UTC m=+3232.561523249"
Oct 31 00:00:46.674848 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.674823    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x"]
Oct 31 00:00:46.719477 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.719446    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x"
Oct 31 00:00:46.731057 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.731029    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x"]
Oct 31 00:00:46.756153 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.756112    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-v2k6x\" (UID: \"da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x"
Oct 31 00:00:46.756317 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.756159    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlwkq\" (UniqueName: \"kubernetes.io/projected/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-kube-api-access-wlwkq\") pod \"maas-api-revoke-sub-ec843018-v2k6x\" (UID: \"da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x"
Oct 31 00:00:46.756317 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.756193    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-v2k6x\" (UID: \"da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x"
Oct 31 00:00:46.778543 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.778388    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:00:46.778543 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.778388    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:00:46.778840 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.778564    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:00:46.824709 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.824668    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:00:46.857393 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.857361    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-v2k6x\" (UID: \"da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x"
Oct 31 00:00:46.857393 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.857397    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlwkq\" (UniqueName: \"kubernetes.io/projected/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-kube-api-access-wlwkq\") pod \"maas-api-revoke-sub-ec843018-v2k6x\" (UID: \"da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x"
Oct 31 00:00:46.857664 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.857427    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-v2k6x\" (UID: \"da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x"
Oct 31 00:00:46.858150 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.858125    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-v2k6x\" (UID: \"da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x"
Oct 31 00:00:46.860096 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.860067    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-v2k6x\" (UID: \"da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x"
Oct 31 00:00:46.915396 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:46.915361    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlwkq\" (UniqueName: \"kubernetes.io/projected/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-kube-api-access-wlwkq\") pod \"maas-api-revoke-sub-ec843018-v2k6x\" (UID: \"da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x"
Oct 31 00:00:47.034476 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:47.034386    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x"
Oct 31 00:00:47.271448 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:47.271410    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4g4vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 00:00:47.271678 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:47.271479    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj" podUID="9af70286-d197-473c-be7d-9333a83d1c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 00:00:47.321317 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:47.321280    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-659rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 31 00:00:47.321735 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:47.321346    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv" podUID="c9e4b867-ea81-4fc3-91c4-ce1129b13d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 31 00:00:47.411571 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:47.411524    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x"]
Oct 31 00:00:47.420574 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:00:47.420535    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9a6c60_f4f2_4b3b_90ad_66bba07c5b3b.slice/crio-ddeae797e5dc927f19ce7ad2fed4bab7575ccfbfc995a25065fa38024214b083 WatchSource:0}: Error finding container ddeae797e5dc927f19ce7ad2fed4bab7575ccfbfc995a25065fa38024214b083: Status 404 returned error can't find the container with id ddeae797e5dc927f19ce7ad2fed4bab7575ccfbfc995a25065fa38024214b083
Oct 31 00:00:47.545139 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:47.545100    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x" event={"ID":"da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b","Type":"ContainerStarted","Data":"ddeae797e5dc927f19ce7ad2fed4bab7575ccfbfc995a25065fa38024214b083"}
Oct 31 00:00:47.594850 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:00:47.594765    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:00:48.271162 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:48.271132    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4g4vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused" start-of-body=
Oct 31 00:00:48.271304 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:48.271184    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj" podUID="9af70286-d197-473c-be7d-9333a83d1c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": dial tcp 10.133.1.7:15021: connect: connection refused"
Oct 31 00:00:48.321935 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:48.321900    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-659rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body=
Oct 31 00:00:48.322394 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:48.321953    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv" podUID="c9e4b867-ea81-4fc3-91c4-ce1129b13d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused"
Oct 31 00:00:48.554880 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:48.554846    5078 generic.go:356] "Generic (PLEG): container finished" podID="da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b" containerID="5cb13ea0463b913ebbff2573b19a3a8872e04bcafafed80d9cddb759cc0b6d22" exitCode=0
Oct 31 00:00:48.555103 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:48.554908    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x" event={"ID":"da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b","Type":"ContainerDied","Data":"5cb13ea0463b913ebbff2573b19a3a8872e04bcafafed80d9cddb759cc0b6d22"}
Oct 31 00:00:49.693821 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:49.693793    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x"
Oct 31 00:00:49.792795 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:49.792761    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-maas-api-openshift-ca\") pod \"da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b\" (UID: \"da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b\") "
Oct 31 00:00:49.792996 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:49.792867    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-maas-api-xks-ca\") pod \"da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b\" (UID: \"da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b\") "
Oct 31 00:00:49.792996 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:49.792949    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-kube-api-access-wlwkq\" (UniqueName: \"kubernetes.io/projected/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-kube-api-access-wlwkq\") pod \"da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b\" (UID: \"da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b\") "
Oct 31 00:00:49.793198 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:49.793158    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-maas-api-openshift-ca" pod "da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b" (UID: "da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:00:49.795172 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:49.795139    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-maas-api-xks-ca" pod "da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b" (UID: "da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:00:49.795305 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:49.795251    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-kube-api-access-wlwkq" pod "da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b" (UID: "da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b"). InnerVolumeSpecName "kube-api-access-wlwkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:00:49.894057 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:49.893976    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:00:49.894057 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:49.894017    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlwkq\" (UniqueName: \"kubernetes.io/projected/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-kube-api-access-wlwkq\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:00:49.894057 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:49.894038    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:00:50.312824 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:50.312795    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4g4vj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:00:50.313251 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:50.313220    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj" podUID="9af70286-d197-473c-be7d-9333a83d1c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:00:50.362782 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:50.362736    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-659rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:00:50.362947 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:50.362794    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv" podUID="c9e4b867-ea81-4fc3-91c4-ce1129b13d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:00:50.568719 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:50.568627    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x" event={"ID":"da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b","Type":"ContainerDied","Data":"ddeae797e5dc927f19ce7ad2fed4bab7575ccfbfc995a25065fa38024214b083"}
Oct 31 00:00:50.568719 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:50.568693    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddeae797e5dc927f19ce7ad2fed4bab7575ccfbfc995a25065fa38024214b083"
Oct 31 00:00:50.569023 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:50.568990    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x"
Oct 31 00:00:51.205835 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:51.205793    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:51.206358 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:51.206234    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:51.207139 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:51.207115    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj"
Oct 31 00:00:51.404800 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:51.404739    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-659rv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:00:51.405021 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:51.404805    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv" podUID="c9e4b867-ea81-4fc3-91c4-ce1129b13d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:00:51.661253 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:51.661222    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:51.661548 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:51.661527    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:00:51.662303 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:00:51.662284    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv"
Oct 31 00:01:04.018340 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:04.018267    5078 scope.go:122] "RemoveContainer" containerID="ffd748afaddada7c17c514a1825e2dac966b9c1ec254e9c1857162658854f4d0"
Oct 31 00:01:04.027213 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:04.027190    5078 scope.go:122] "RemoveContainer" containerID="3e2f8b507638e2e6108c4cb529dfb38ba0f06cfe617e542f54427f693065d812"
Oct 31 00:01:06.734622 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:06.733688    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-42wvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:01:06.734622 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:06.733766    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb" podUID="c76a2fb3-5cc9-4e1d-bc93-c8acf8723819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:01:06.753746 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:06.753710    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4662g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:01:06.754032 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:06.754004    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g" podUID="9a60053e-3454-4ef1-9b40-9194f19b840d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:01:10.045549 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:10.045515    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh"]
Oct 31 00:01:10.095244 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:10.095216    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5d4f234-x8tnh"]
Oct 31 00:01:10.302014 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:10.301935    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d292e2d-e61c-4140-a6e6-fe316625d549" path="/var/lib/kubelet/pods/8d292e2d-e61c-4140-a6e6-fe316625d549/volumes"
Oct 31 00:01:17.589652 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:01:17.589616    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:01:22.171558 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.171518    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx"]
Oct 31 00:01:22.173979 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.171853    5078 cpu_manager.go:414] "containerMap: removing container" podUID="da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b" containerName="revoke-keys"
Oct 31 00:01:22.173979 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.171864    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b" containerName="revoke-keys"
Oct 31 00:01:22.173979 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.171921    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b" containerName="revoke-keys"
Oct 31 00:01:22.174797 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.174779    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx"
Oct 31 00:01:22.278862 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.278829    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:01:22.278862 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.278838    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:01:22.279122 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.278932    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:01:22.279122 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.278952    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:01:22.280822 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.280797    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx"]
Oct 31 00:01:22.310987 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.310958    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fsl5\" (UniqueName: \"kubernetes.io/projected/61d969ae-4db2-4b2c-a00c-f8de742bf25d-kube-api-access-5fsl5\") pod \"maas-api-revoke-sub-046e1343-4rhdx\" (UID: \"61d969ae-4db2-4b2c-a00c-f8de742bf25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx"
Oct 31 00:01:22.311115 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.311009    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d969ae-4db2-4b2c-a00c-f8de742bf25d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-4rhdx\" (UID: \"61d969ae-4db2-4b2c-a00c-f8de742bf25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx"
Oct 31 00:01:22.311160 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.311138    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d969ae-4db2-4b2c-a00c-f8de742bf25d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-4rhdx\" (UID: \"61d969ae-4db2-4b2c-a00c-f8de742bf25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx"
Oct 31 00:01:22.411574 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.411543    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d969ae-4db2-4b2c-a00c-f8de742bf25d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-4rhdx\" (UID: \"61d969ae-4db2-4b2c-a00c-f8de742bf25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx"
Oct 31 00:01:22.411728 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.411621    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d969ae-4db2-4b2c-a00c-f8de742bf25d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-4rhdx\" (UID: \"61d969ae-4db2-4b2c-a00c-f8de742bf25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx"
Oct 31 00:01:22.411728 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.411665    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fsl5\" (UniqueName: \"kubernetes.io/projected/61d969ae-4db2-4b2c-a00c-f8de742bf25d-kube-api-access-5fsl5\") pod \"maas-api-revoke-sub-046e1343-4rhdx\" (UID: \"61d969ae-4db2-4b2c-a00c-f8de742bf25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx"
Oct 31 00:01:22.412346 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.412268    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d969ae-4db2-4b2c-a00c-f8de742bf25d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-4rhdx\" (UID: \"61d969ae-4db2-4b2c-a00c-f8de742bf25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx"
Oct 31 00:01:22.413970 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.413952    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d969ae-4db2-4b2c-a00c-f8de742bf25d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-4rhdx\" (UID: \"61d969ae-4db2-4b2c-a00c-f8de742bf25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx"
Oct 31 00:01:22.468077 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.468017    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fsl5\" (UniqueName: \"kubernetes.io/projected/61d969ae-4db2-4b2c-a00c-f8de742bf25d-kube-api-access-5fsl5\") pod \"maas-api-revoke-sub-046e1343-4rhdx\" (UID: \"61d969ae-4db2-4b2c-a00c-f8de742bf25d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx"
Oct 31 00:01:22.486233 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.486210    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx"
Oct 31 00:01:22.854450 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:22.854417    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx"]
Oct 31 00:01:22.857327 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:01:22.857299    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d969ae_4db2_4b2c_a00c_f8de742bf25d.slice/crio-e5252b483a59beb3c89f7bc6052c81715d29169496c2b3f4a22c29364aaecaba WatchSource:0}: Error finding container e5252b483a59beb3c89f7bc6052c81715d29169496c2b3f4a22c29364aaecaba: Status 404 returned error can't find the container with id e5252b483a59beb3c89f7bc6052c81715d29169496c2b3f4a22c29364aaecaba
Oct 31 00:01:23.744814 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:23.744787    5078 generic.go:356] "Generic (PLEG): container finished" podID="61d969ae-4db2-4b2c-a00c-f8de742bf25d" containerID="6d2e110c60b0fff7ef0548162007c8b5638348a97625400ac2f73aa35fd7fdcf" exitCode=0
Oct 31 00:01:23.745136 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:23.744828    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx" event={"ID":"61d969ae-4db2-4b2c-a00c-f8de742bf25d","Type":"ContainerDied","Data":"6d2e110c60b0fff7ef0548162007c8b5638348a97625400ac2f73aa35fd7fdcf"}
Oct 31 00:01:23.745136 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:23.744884    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx" event={"ID":"61d969ae-4db2-4b2c-a00c-f8de742bf25d","Type":"ContainerStarted","Data":"e5252b483a59beb3c89f7bc6052c81715d29169496c2b3f4a22c29364aaecaba"}
Oct 31 00:01:24.045021 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:24.044972    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl"]
Oct 31 00:01:24.046405 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:24.046387    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-926719d0-hwhvl"]
Oct 31 00:01:24.297298 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:24.297209    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45326690-de30-4763-9a1e-b9ca57da55bc" path="/var/lib/kubelet/pods/45326690-de30-4763-9a1e-b9ca57da55bc/volumes"
Oct 31 00:01:24.873381 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:24.873359    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx"
Oct 31 00:01:25.033024 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:25.032953    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d969ae-4db2-4b2c-a00c-f8de742bf25d-kube-api-access-5fsl5\" (UniqueName: \"kubernetes.io/projected/61d969ae-4db2-4b2c-a00c-f8de742bf25d-kube-api-access-5fsl5\") pod \"61d969ae-4db2-4b2c-a00c-f8de742bf25d\" (UID: \"61d969ae-4db2-4b2c-a00c-f8de742bf25d\") "
Oct 31 00:01:25.033201 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:25.033094    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d969ae-4db2-4b2c-a00c-f8de742bf25d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d969ae-4db2-4b2c-a00c-f8de742bf25d-maas-api-xks-ca\") pod \"61d969ae-4db2-4b2c-a00c-f8de742bf25d\" (UID: \"61d969ae-4db2-4b2c-a00c-f8de742bf25d\") "
Oct 31 00:01:25.033201 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:25.033117    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d969ae-4db2-4b2c-a00c-f8de742bf25d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d969ae-4db2-4b2c-a00c-f8de742bf25d-maas-api-openshift-ca\") pod \"61d969ae-4db2-4b2c-a00c-f8de742bf25d\" (UID: \"61d969ae-4db2-4b2c-a00c-f8de742bf25d\") "
Oct 31 00:01:25.033469 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:25.033446    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d969ae-4db2-4b2c-a00c-f8de742bf25d-maas-api-openshift-ca" pod "61d969ae-4db2-4b2c-a00c-f8de742bf25d" (UID: "61d969ae-4db2-4b2c-a00c-f8de742bf25d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:01:25.035246 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:25.035227    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d969ae-4db2-4b2c-a00c-f8de742bf25d-kube-api-access-5fsl5" pod "61d969ae-4db2-4b2c-a00c-f8de742bf25d" (UID: "61d969ae-4db2-4b2c-a00c-f8de742bf25d"). InnerVolumeSpecName "kube-api-access-5fsl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:01:25.035373 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:25.035358    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d969ae-4db2-4b2c-a00c-f8de742bf25d-maas-api-xks-ca" pod "61d969ae-4db2-4b2c-a00c-f8de742bf25d" (UID: "61d969ae-4db2-4b2c-a00c-f8de742bf25d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:01:25.134554 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:25.134528    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61d969ae-4db2-4b2c-a00c-f8de742bf25d-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:01:25.134554 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:25.134553    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61d969ae-4db2-4b2c-a00c-f8de742bf25d-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:01:25.134740 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:25.134563    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fsl5\" (UniqueName: \"kubernetes.io/projected/61d969ae-4db2-4b2c-a00c-f8de742bf25d-kube-api-access-5fsl5\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:01:25.755294 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:25.755259    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx" event={"ID":"61d969ae-4db2-4b2c-a00c-f8de742bf25d","Type":"ContainerDied","Data":"e5252b483a59beb3c89f7bc6052c81715d29169496c2b3f4a22c29364aaecaba"}
Oct 31 00:01:25.755481 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:25.755304    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5252b483a59beb3c89f7bc6052c81715d29169496c2b3f4a22c29364aaecaba"
Oct 31 00:01:25.755481 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:25.755300    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx"
Oct 31 00:01:32.961376 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:32.961339    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9"]
Oct 31 00:01:32.961797 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:32.961695    5078 cpu_manager.go:414] "containerMap: removing container" podUID="61d969ae-4db2-4b2c-a00c-f8de742bf25d" containerName="revoke-keys"
Oct 31 00:01:32.961797 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:32.961706    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d969ae-4db2-4b2c-a00c-f8de742bf25d" containerName="revoke-keys"
Oct 31 00:01:32.961797 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:32.961774    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d969ae-4db2-4b2c-a00c-f8de742bf25d" containerName="revoke-keys"
Oct 31 00:01:32.964735 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:32.964715    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9"
Oct 31 00:01:33.019881 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:33.019850    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:01:33.068544 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:33.068510    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:01:33.068544 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:33.068512    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:01:33.068788 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:33.068569    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:01:33.070241 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:33.070219    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9"]
Oct 31 00:01:33.105673 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:33.105643    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-kj6g9\" (UID: \"f25ff8ed-973b-4c27-a0be-19b9fbd30f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9"
Oct 31 00:01:33.105822 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:33.105769    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-kj6g9\" (UID: \"f25ff8ed-973b-4c27-a0be-19b9fbd30f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9"
Oct 31 00:01:33.105822 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:33.105815    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt5pn\" (UniqueName: \"kubernetes.io/projected/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-kube-api-access-bt5pn\") pod \"maas-api-revoke-sub-d3fba056-kj6g9\" (UID: \"f25ff8ed-973b-4c27-a0be-19b9fbd30f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9"
Oct 31 00:01:33.206896 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:33.206865    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-kj6g9\" (UID: \"f25ff8ed-973b-4c27-a0be-19b9fbd30f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9"
Oct 31 00:01:33.207089 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:33.206904    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5pn\" (UniqueName: \"kubernetes.io/projected/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-kube-api-access-bt5pn\") pod \"maas-api-revoke-sub-d3fba056-kj6g9\" (UID: \"f25ff8ed-973b-4c27-a0be-19b9fbd30f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9"
Oct 31 00:01:33.207089 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:33.207042    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-kj6g9\" (UID: \"f25ff8ed-973b-4c27-a0be-19b9fbd30f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9"
Oct 31 00:01:33.207584 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:33.207561    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-kj6g9\" (UID: \"f25ff8ed-973b-4c27-a0be-19b9fbd30f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9"
Oct 31 00:01:33.209515 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:33.209492    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-kj6g9\" (UID: \"f25ff8ed-973b-4c27-a0be-19b9fbd30f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9"
Oct 31 00:01:33.263863 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:33.263798    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt5pn\" (UniqueName: \"kubernetes.io/projected/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-kube-api-access-bt5pn\") pod \"maas-api-revoke-sub-d3fba056-kj6g9\" (UID: \"f25ff8ed-973b-4c27-a0be-19b9fbd30f9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9"
Oct 31 00:01:33.276732 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:33.276711    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9"
Oct 31 00:01:33.645363 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:33.645338    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9"]
Oct 31 00:01:33.647862 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:01:33.647837    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25ff8ed_973b_4c27_a0be_19b9fbd30f9d.slice/crio-62920f43b9055e1abf7d6a09251636cc25ef035e0aa47ebe2fce223a15a75d92 WatchSource:0}: Error finding container 62920f43b9055e1abf7d6a09251636cc25ef035e0aa47ebe2fce223a15a75d92: Status 404 returned error can't find the container with id 62920f43b9055e1abf7d6a09251636cc25ef035e0aa47ebe2fce223a15a75d92
Oct 31 00:01:33.794015 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:33.793982    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9" event={"ID":"f25ff8ed-973b-4c27-a0be-19b9fbd30f9d","Type":"ContainerStarted","Data":"62920f43b9055e1abf7d6a09251636cc25ef035e0aa47ebe2fce223a15a75d92"}
Oct 31 00:01:34.799991 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:34.799961    5078 generic.go:356] "Generic (PLEG): container finished" podID="f25ff8ed-973b-4c27-a0be-19b9fbd30f9d" containerID="558674bb1f65fa341136b4bc9195bf2e79fe8005dc65082ba24966a23c3d0d89" exitCode=0
Oct 31 00:01:34.800380 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:34.800007    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9" event={"ID":"f25ff8ed-973b-4c27-a0be-19b9fbd30f9d","Type":"ContainerDied","Data":"558674bb1f65fa341136b4bc9195bf2e79fe8005dc65082ba24966a23c3d0d89"}
Oct 31 00:01:35.983923 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:35.983894    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9"
Oct 31 00:01:36.132120 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:36.132009    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-maas-api-openshift-ca\") pod \"f25ff8ed-973b-4c27-a0be-19b9fbd30f9d\" (UID: \"f25ff8ed-973b-4c27-a0be-19b9fbd30f9d\") "
Oct 31 00:01:36.132314 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:36.132150    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-maas-api-xks-ca\") pod \"f25ff8ed-973b-4c27-a0be-19b9fbd30f9d\" (UID: \"f25ff8ed-973b-4c27-a0be-19b9fbd30f9d\") "
Oct 31 00:01:36.132314 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:36.132235    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-kube-api-access-bt5pn\" (UniqueName: \"kubernetes.io/projected/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-kube-api-access-bt5pn\") pod \"f25ff8ed-973b-4c27-a0be-19b9fbd30f9d\" (UID: \"f25ff8ed-973b-4c27-a0be-19b9fbd30f9d\") "
Oct 31 00:01:36.132906 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:36.132859    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-maas-api-openshift-ca" pod "f25ff8ed-973b-4c27-a0be-19b9fbd30f9d" (UID: "f25ff8ed-973b-4c27-a0be-19b9fbd30f9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:01:36.135883 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:36.135826    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-kube-api-access-bt5pn" pod "f25ff8ed-973b-4c27-a0be-19b9fbd30f9d" (UID: "f25ff8ed-973b-4c27-a0be-19b9fbd30f9d"). InnerVolumeSpecName "kube-api-access-bt5pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:01:36.136112 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:36.136082    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-maas-api-xks-ca" pod "f25ff8ed-973b-4c27-a0be-19b9fbd30f9d" (UID: "f25ff8ed-973b-4c27-a0be-19b9fbd30f9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:01:36.233341 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:36.233290    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:01:36.233341 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:36.233334    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:01:36.233341 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:36.233351    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt5pn\" (UniqueName: \"kubernetes.io/projected/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d-kube-api-access-bt5pn\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:01:36.829763 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:36.828724    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9" event={"ID":"f25ff8ed-973b-4c27-a0be-19b9fbd30f9d","Type":"ContainerDied","Data":"62920f43b9055e1abf7d6a09251636cc25ef035e0aa47ebe2fce223a15a75d92"}
Oct 31 00:01:36.829763 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:36.828799    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62920f43b9055e1abf7d6a09251636cc25ef035e0aa47ebe2fce223a15a75d92"
Oct 31 00:01:36.829763 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:36.828881    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9"
Oct 31 00:01:37.713621 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:37.712673    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-659rv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:01:37.713621 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:01:37.713211    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-659rv" podUID="c9e4b867-ea81-4fc3-91c4-ce1129b13d3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:01:47.589521 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:01:47.589485    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:02:04.051233 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:04.051204    5078 scope.go:122] "RemoveContainer" containerID="60dfcbc499a57a0238d7d00eaae75af77cc046c0b16f1ca32ed5b119fdf6feb4"
Oct 31 00:02:04.059619 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:04.059580    5078 scope.go:122] "RemoveContainer" containerID="4b6d442a0cb5d72f65d7be987b400981d62edc1e82a7f940cd946016f34ebd4b"
Oct 31 00:02:10.733404 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:10.733370    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn"]
Oct 31 00:02:10.733834 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:10.733710    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f25ff8ed-973b-4c27-a0be-19b9fbd30f9d" containerName="revoke-keys"
Oct 31 00:02:10.733834 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:10.733722    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25ff8ed-973b-4c27-a0be-19b9fbd30f9d" containerName="revoke-keys"
Oct 31 00:02:10.733834 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:10.733799    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f25ff8ed-973b-4c27-a0be-19b9fbd30f9d" containerName="revoke-keys"
Oct 31 00:02:10.738437 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:10.738410    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn"
Oct 31 00:02:10.794640 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:10.794581    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn"]
Oct 31 00:02:10.841606 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:10.841566    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:02:10.841801 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:10.841633    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:02:10.841801 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:10.841633    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:02:10.841938 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:10.841832    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:02:10.916040 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:10.916005    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea35cc93-a1e8-4a3f-93b5-2708a5027762-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2rcqn\" (UID: \"ea35cc93-a1e8-4a3f-93b5-2708a5027762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn"
Oct 31 00:02:10.916249 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:10.916160    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea35cc93-a1e8-4a3f-93b5-2708a5027762-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2rcqn\" (UID: \"ea35cc93-a1e8-4a3f-93b5-2708a5027762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn"
Oct 31 00:02:10.916249 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:10.916219    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb5lv\" (UniqueName: \"kubernetes.io/projected/ea35cc93-a1e8-4a3f-93b5-2708a5027762-kube-api-access-pb5lv\") pod \"maas-api-revoke-sub-ae594979-2rcqn\" (UID: \"ea35cc93-a1e8-4a3f-93b5-2708a5027762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn"
Oct 31 00:02:11.017093 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:11.017015    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb5lv\" (UniqueName: \"kubernetes.io/projected/ea35cc93-a1e8-4a3f-93b5-2708a5027762-kube-api-access-pb5lv\") pod \"maas-api-revoke-sub-ae594979-2rcqn\" (UID: \"ea35cc93-a1e8-4a3f-93b5-2708a5027762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn"
Oct 31 00:02:11.017093 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:11.017053    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea35cc93-a1e8-4a3f-93b5-2708a5027762-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2rcqn\" (UID: \"ea35cc93-a1e8-4a3f-93b5-2708a5027762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn"
Oct 31 00:02:11.017349 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:11.017125    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea35cc93-a1e8-4a3f-93b5-2708a5027762-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2rcqn\" (UID: \"ea35cc93-a1e8-4a3f-93b5-2708a5027762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn"
Oct 31 00:02:11.017739 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:11.017720    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea35cc93-a1e8-4a3f-93b5-2708a5027762-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-2rcqn\" (UID: \"ea35cc93-a1e8-4a3f-93b5-2708a5027762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn"
Oct 31 00:02:11.019505 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:11.019484    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea35cc93-a1e8-4a3f-93b5-2708a5027762-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-2rcqn\" (UID: \"ea35cc93-a1e8-4a3f-93b5-2708a5027762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn"
Oct 31 00:02:11.075574 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:11.075544    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb5lv\" (UniqueName: \"kubernetes.io/projected/ea35cc93-a1e8-4a3f-93b5-2708a5027762-kube-api-access-pb5lv\") pod \"maas-api-revoke-sub-ae594979-2rcqn\" (UID: \"ea35cc93-a1e8-4a3f-93b5-2708a5027762\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn"
Oct 31 00:02:11.355168 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:11.355140    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn"
Oct 31 00:02:11.728820 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:11.728791    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn"]
Oct 31 00:02:11.730847 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:02:11.730824    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea35cc93_a1e8_4a3f_93b5_2708a5027762.slice/crio-5bfef5a0d7b3794184071da93b41a23c1c6edffd527eecb991681c7575045b51 WatchSource:0}: Error finding container 5bfef5a0d7b3794184071da93b41a23c1c6edffd527eecb991681c7575045b51: Status 404 returned error can't find the container with id 5bfef5a0d7b3794184071da93b41a23c1c6edffd527eecb991681c7575045b51
Oct 31 00:02:12.022914 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:12.022827    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn" event={"ID":"ea35cc93-a1e8-4a3f-93b5-2708a5027762","Type":"ContainerStarted","Data":"6a8c582db785030bde85b4653f04ee1ad6a7f96f21620590bcbb8da0792539f2"}
Oct 31 00:02:12.022914 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:12.022894    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn" event={"ID":"ea35cc93-a1e8-4a3f-93b5-2708a5027762","Type":"ContainerStarted","Data":"5bfef5a0d7b3794184071da93b41a23c1c6edffd527eecb991681c7575045b51"}
Oct 31 00:02:12.140019 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:12.139961    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn" podStartSLOduration=2.139943245 podStartE2EDuration="2.139943245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:12.137302518 +0000 UTC m=+3318.025208883" watchObservedRunningTime="2026-10-31 00:02:12.139943245 +0000 UTC m=+3318.027849599"
Oct 31 00:02:13.029911 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:13.029875    5078 generic.go:356] "Generic (PLEG): container finished" podID="ea35cc93-a1e8-4a3f-93b5-2708a5027762" containerID="6a8c582db785030bde85b4653f04ee1ad6a7f96f21620590bcbb8da0792539f2" exitCode=0
Oct 31 00:02:13.030334 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:13.029956    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn" event={"ID":"ea35cc93-a1e8-4a3f-93b5-2708a5027762","Type":"ContainerDied","Data":"6a8c582db785030bde85b4653f04ee1ad6a7f96f21620590bcbb8da0792539f2"}
Oct 31 00:02:14.161126 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:14.161103    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn"
Oct 31 00:02:14.344886 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:14.344850    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea35cc93-a1e8-4a3f-93b5-2708a5027762-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea35cc93-a1e8-4a3f-93b5-2708a5027762-maas-api-openshift-ca\") pod \"ea35cc93-a1e8-4a3f-93b5-2708a5027762\" (UID: \"ea35cc93-a1e8-4a3f-93b5-2708a5027762\") "
Oct 31 00:02:14.345092 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:14.344904    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea35cc93-a1e8-4a3f-93b5-2708a5027762-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea35cc93-a1e8-4a3f-93b5-2708a5027762-maas-api-xks-ca\") pod \"ea35cc93-a1e8-4a3f-93b5-2708a5027762\" (UID: \"ea35cc93-a1e8-4a3f-93b5-2708a5027762\") "
Oct 31 00:02:14.345092 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:14.344931    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea35cc93-a1e8-4a3f-93b5-2708a5027762-kube-api-access-pb5lv\" (UniqueName: \"kubernetes.io/projected/ea35cc93-a1e8-4a3f-93b5-2708a5027762-kube-api-access-pb5lv\") pod \"ea35cc93-a1e8-4a3f-93b5-2708a5027762\" (UID: \"ea35cc93-a1e8-4a3f-93b5-2708a5027762\") "
Oct 31 00:02:14.345279 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:14.345256    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea35cc93-a1e8-4a3f-93b5-2708a5027762-maas-api-openshift-ca" pod "ea35cc93-a1e8-4a3f-93b5-2708a5027762" (UID: "ea35cc93-a1e8-4a3f-93b5-2708a5027762"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:02:14.347187 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:14.347164    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea35cc93-a1e8-4a3f-93b5-2708a5027762-kube-api-access-pb5lv" pod "ea35cc93-a1e8-4a3f-93b5-2708a5027762" (UID: "ea35cc93-a1e8-4a3f-93b5-2708a5027762"). InnerVolumeSpecName "kube-api-access-pb5lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:02:14.347327 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:14.347306    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea35cc93-a1e8-4a3f-93b5-2708a5027762-maas-api-xks-ca" pod "ea35cc93-a1e8-4a3f-93b5-2708a5027762" (UID: "ea35cc93-a1e8-4a3f-93b5-2708a5027762"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:02:14.445797 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:14.445769    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea35cc93-a1e8-4a3f-93b5-2708a5027762-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:02:14.445797 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:14.445795    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb5lv\" (UniqueName: \"kubernetes.io/projected/ea35cc93-a1e8-4a3f-93b5-2708a5027762-kube-api-access-pb5lv\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:02:14.445992 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:14.445805    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea35cc93-a1e8-4a3f-93b5-2708a5027762-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:02:15.040370 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:15.040337    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn" event={"ID":"ea35cc93-a1e8-4a3f-93b5-2708a5027762","Type":"ContainerDied","Data":"5bfef5a0d7b3794184071da93b41a23c1c6edffd527eecb991681c7575045b51"}
Oct 31 00:02:15.040555 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:15.040373    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn"
Oct 31 00:02:15.040555 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:15.040383    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bfef5a0d7b3794184071da93b41a23c1c6edffd527eecb991681c7575045b51"
Oct 31 00:02:17.589311 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:02:17.589276    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:02:38.722677 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.722576    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2"]
Oct 31 00:02:38.723172 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.722933    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ea35cc93-a1e8-4a3f-93b5-2708a5027762" containerName="revoke-keys"
Oct 31 00:02:38.723172 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.722945    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea35cc93-a1e8-4a3f-93b5-2708a5027762" containerName="revoke-keys"
Oct 31 00:02:38.723172 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.723021    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea35cc93-a1e8-4a3f-93b5-2708a5027762" containerName="revoke-keys"
Oct 31 00:02:38.726022 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.726005    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2"
Oct 31 00:02:38.798143 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.798102    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10528c4-c8fd-41a1-b028-a82e08997577-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-blzg2\" (UID: \"f10528c4-c8fd-41a1-b028-a82e08997577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2"
Oct 31 00:02:38.798339 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.798188    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2lxc\" (UniqueName: \"kubernetes.io/projected/f10528c4-c8fd-41a1-b028-a82e08997577-kube-api-access-j2lxc\") pod \"maas-api-revoke-sub-a4aef7d7-blzg2\" (UID: \"f10528c4-c8fd-41a1-b028-a82e08997577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2"
Oct 31 00:02:38.798420 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.798337    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10528c4-c8fd-41a1-b028-a82e08997577-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-blzg2\" (UID: \"f10528c4-c8fd-41a1-b028-a82e08997577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2"
Oct 31 00:02:38.829315 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.829281    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:02:38.829315 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.829281    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:02:38.829315 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.829320    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:02:38.829315 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.829290    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:02:38.831240 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.831215    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2"]
Oct 31 00:02:38.898885 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.898857    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10528c4-c8fd-41a1-b028-a82e08997577-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-blzg2\" (UID: \"f10528c4-c8fd-41a1-b028-a82e08997577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2"
Oct 31 00:02:38.898885 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.898888    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2lxc\" (UniqueName: \"kubernetes.io/projected/f10528c4-c8fd-41a1-b028-a82e08997577-kube-api-access-j2lxc\") pod \"maas-api-revoke-sub-a4aef7d7-blzg2\" (UID: \"f10528c4-c8fd-41a1-b028-a82e08997577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2"
Oct 31 00:02:38.899158 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.898911    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10528c4-c8fd-41a1-b028-a82e08997577-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-blzg2\" (UID: \"f10528c4-c8fd-41a1-b028-a82e08997577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2"
Oct 31 00:02:38.899548 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.899512    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10528c4-c8fd-41a1-b028-a82e08997577-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-blzg2\" (UID: \"f10528c4-c8fd-41a1-b028-a82e08997577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2"
Oct 31 00:02:38.901469 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.901445    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10528c4-c8fd-41a1-b028-a82e08997577-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-blzg2\" (UID: \"f10528c4-c8fd-41a1-b028-a82e08997577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2"
Oct 31 00:02:38.957105 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:38.957066    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2lxc\" (UniqueName: \"kubernetes.io/projected/f10528c4-c8fd-41a1-b028-a82e08997577-kube-api-access-j2lxc\") pod \"maas-api-revoke-sub-a4aef7d7-blzg2\" (UID: \"f10528c4-c8fd-41a1-b028-a82e08997577\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2"
Oct 31 00:02:39.038859 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:39.038782    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2"
Oct 31 00:02:39.049378 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:39.049353    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r"]
Oct 31 00:02:39.056352 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:39.056319    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r"]
Oct 31 00:02:39.058208 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:39.058187    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b150622-sbc6r"]
Oct 31 00:02:39.059665 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:39.059643    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eaea140b-2nf2r"]
Oct 31 00:02:39.413125 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:39.413093    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2"]
Oct 31 00:02:39.414917 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:02:39.414889    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10528c4_c8fd_41a1_b028_a82e08997577.slice/crio-9e150d08328cf69f531ae429653a4e316a1991dba7e4293352fd257e829458f2 WatchSource:0}: Error finding container 9e150d08328cf69f531ae429653a4e316a1991dba7e4293352fd257e829458f2: Status 404 returned error can't find the container with id 9e150d08328cf69f531ae429653a4e316a1991dba7e4293352fd257e829458f2
Oct 31 00:02:40.182053 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:40.182015    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2" event={"ID":"f10528c4-c8fd-41a1-b028-a82e08997577","Type":"ContainerStarted","Data":"ae4674ce103504dcb1ec48aa783a1a0a5bc0303dbcd9321eeee0e14b4500c0a3"}
Oct 31 00:02:40.182456 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:40.182079    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2" event={"ID":"f10528c4-c8fd-41a1-b028-a82e08997577","Type":"ContainerStarted","Data":"9e150d08328cf69f531ae429653a4e316a1991dba7e4293352fd257e829458f2"}
Oct 31 00:02:40.295875 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:40.295843    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec9d021b-6fa0-4df4-8021-14ab08a20c3e" path="/var/lib/kubelet/pods/ec9d021b-6fa0-4df4-8021-14ab08a20c3e/volumes"
Oct 31 00:02:40.296256 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:40.296244    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f189cd0f-f06e-4495-931c-38aac8abb98a" path="/var/lib/kubelet/pods/f189cd0f-f06e-4495-931c-38aac8abb98a/volumes"
Oct 31 00:02:40.301757 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:40.301714    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2" podStartSLOduration=2.301702459 podStartE2EDuration="2.301702459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:40.29828656 +0000 UTC m=+3346.186192912" watchObservedRunningTime="2026-10-31 00:02:40.301702459 +0000 UTC m=+3346.189608811"
Oct 31 00:02:41.188424 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:41.188390    5078 generic.go:356] "Generic (PLEG): container finished" podID="f10528c4-c8fd-41a1-b028-a82e08997577" containerID="ae4674ce103504dcb1ec48aa783a1a0a5bc0303dbcd9321eeee0e14b4500c0a3" exitCode=0
Oct 31 00:02:41.188860 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:41.188464    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2" event={"ID":"f10528c4-c8fd-41a1-b028-a82e08997577","Type":"ContainerDied","Data":"ae4674ce103504dcb1ec48aa783a1a0a5bc0303dbcd9321eeee0e14b4500c0a3"}
Oct 31 00:02:42.326032 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:42.326005    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2"
Oct 31 00:02:42.432340 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:42.432304    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10528c4-c8fd-41a1-b028-a82e08997577-kube-api-access-j2lxc\" (UniqueName: \"kubernetes.io/projected/f10528c4-c8fd-41a1-b028-a82e08997577-kube-api-access-j2lxc\") pod \"f10528c4-c8fd-41a1-b028-a82e08997577\" (UID: \"f10528c4-c8fd-41a1-b028-a82e08997577\") "
Oct 31 00:02:42.432340 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:42.432340    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10528c4-c8fd-41a1-b028-a82e08997577-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10528c4-c8fd-41a1-b028-a82e08997577-maas-api-openshift-ca\") pod \"f10528c4-c8fd-41a1-b028-a82e08997577\" (UID: \"f10528c4-c8fd-41a1-b028-a82e08997577\") "
Oct 31 00:02:42.432573 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:42.432386    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f10528c4-c8fd-41a1-b028-a82e08997577-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10528c4-c8fd-41a1-b028-a82e08997577-maas-api-xks-ca\") pod \"f10528c4-c8fd-41a1-b028-a82e08997577\" (UID: \"f10528c4-c8fd-41a1-b028-a82e08997577\") "
Oct 31 00:02:42.432721 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:42.432699    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10528c4-c8fd-41a1-b028-a82e08997577-maas-api-openshift-ca" pod "f10528c4-c8fd-41a1-b028-a82e08997577" (UID: "f10528c4-c8fd-41a1-b028-a82e08997577"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:02:42.434683 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:42.434657    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f10528c4-c8fd-41a1-b028-a82e08997577-maas-api-xks-ca" pod "f10528c4-c8fd-41a1-b028-a82e08997577" (UID: "f10528c4-c8fd-41a1-b028-a82e08997577"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:02:42.434825 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:42.434812    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10528c4-c8fd-41a1-b028-a82e08997577-kube-api-access-j2lxc" pod "f10528c4-c8fd-41a1-b028-a82e08997577" (UID: "f10528c4-c8fd-41a1-b028-a82e08997577"). InnerVolumeSpecName "kube-api-access-j2lxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:02:42.533672 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:42.533568    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2lxc\" (UniqueName: \"kubernetes.io/projected/f10528c4-c8fd-41a1-b028-a82e08997577-kube-api-access-j2lxc\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:02:42.533672 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:42.533630    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10528c4-c8fd-41a1-b028-a82e08997577-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:02:42.533672 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:42.533646    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10528c4-c8fd-41a1-b028-a82e08997577-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:02:43.199677 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:43.199646    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2"
Oct 31 00:02:43.199677 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:43.199661    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2" event={"ID":"f10528c4-c8fd-41a1-b028-a82e08997577","Type":"ContainerDied","Data":"9e150d08328cf69f531ae429653a4e316a1991dba7e4293352fd257e829458f2"}
Oct 31 00:02:43.199892 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:02:43.199708    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e150d08328cf69f531ae429653a4e316a1991dba7e4293352fd257e829458f2"
Oct 31 00:02:47.590889 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:02:47.590851    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:03:03.050688 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:03.050650    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2"]
Oct 31 00:03:03.070370 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:03.070333    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bea43a4-bgmp2"]
Oct 31 00:03:04.081826 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:04.081787    5078 scope.go:122] "RemoveContainer" containerID="e0b280e75b6f340e12df7d34d838cff313cf5143072cbf42057188bf9e5d5704"
Oct 31 00:03:04.090676 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:04.090655    5078 scope.go:122] "RemoveContainer" containerID="662b3d9cea8f9ae3e8663bcfde8d665b88242d91583f404a3679f805c9eb46a6"
Oct 31 00:03:04.099083 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:04.099060    5078 scope.go:122] "RemoveContainer" containerID="9a1aa569f3363d7a384380dc6dd9d59b1c81bda575d8a4b23203ee3dc3ee8971"
Oct 31 00:03:04.107091 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:04.107070    5078 scope.go:122] "RemoveContainer" containerID="15ea60ee66c37f0c1370c89cf5e06e8ae256be2b025ff861f144cc6d87b524c5"
Oct 31 00:03:04.114659 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:04.114640    5078 scope.go:122] "RemoveContainer" containerID="36585bf59ee991a99fb6cbcafbbf588182a21cf58163d37b5b7950a95e7dac5e"
Oct 31 00:03:04.122411 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:04.122391    5078 scope.go:122] "RemoveContainer" containerID="3f10eb8c26319b918bd80b97797eca655d5a28f489a80fd0b9195dd3b662da98"
Oct 31 00:03:04.297122 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:04.297086    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="534c4672-90ed-47ce-9a7b-28aac0ccc886" path="/var/lib/kubelet/pods/534c4672-90ed-47ce-9a7b-28aac0ccc886/volumes"
Oct 31 00:03:09.499999 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.499968    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744"]
Oct 31 00:03:09.500420 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.500320    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f10528c4-c8fd-41a1-b028-a82e08997577" containerName="revoke-keys"
Oct 31 00:03:09.500420 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.500331    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10528c4-c8fd-41a1-b028-a82e08997577" containerName="revoke-keys"
Oct 31 00:03:09.500420 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.500398    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10528c4-c8fd-41a1-b028-a82e08997577" containerName="revoke-keys"
Oct 31 00:03:09.505088 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.505069    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744"
Oct 31 00:03:09.561928 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.561889    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744"]
Oct 31 00:03:09.609410 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.609379    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:03:09.609680 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.609663    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:03:09.609778 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.609700    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:03:09.609778 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.609742    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:03:09.630285 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.630257    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gt7n\" (UniqueName: \"kubernetes.io/projected/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-kube-api-access-9gt7n\") pod \"maas-api-revoke-sub-79805ca9-gx744\" (UID: \"dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744"
Oct 31 00:03:09.630447 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.630312    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-gx744\" (UID: \"dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744"
Oct 31 00:03:09.630447 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.630340    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-gx744\" (UID: \"dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744"
Oct 31 00:03:09.731115 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.731075    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-gx744\" (UID: \"dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744"
Oct 31 00:03:09.731343 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.731122    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-gx744\" (UID: \"dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744"
Oct 31 00:03:09.731343 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.731215    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gt7n\" (UniqueName: \"kubernetes.io/projected/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-kube-api-access-9gt7n\") pod \"maas-api-revoke-sub-79805ca9-gx744\" (UID: \"dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744"
Oct 31 00:03:09.731804 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.731782    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-gx744\" (UID: \"dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744"
Oct 31 00:03:09.733723 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.733702    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-gx744\" (UID: \"dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744"
Oct 31 00:03:09.789297 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.789222    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gt7n\" (UniqueName: \"kubernetes.io/projected/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-kube-api-access-9gt7n\") pod \"maas-api-revoke-sub-79805ca9-gx744\" (UID: \"dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744"
Oct 31 00:03:09.817056 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:09.817021    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744"
Oct 31 00:03:10.195248 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:10.195219    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744"]
Oct 31 00:03:10.197196 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:03:10.197167    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcfca9d2_aae8_4f55_9ec2_44e53e9ccae9.slice/crio-568f93aeb59ad92d877d2948cfe550d8e626803f864d5bb0e429a79dc6e0691a WatchSource:0}: Error finding container 568f93aeb59ad92d877d2948cfe550d8e626803f864d5bb0e429a79dc6e0691a: Status 404 returned error can't find the container with id 568f93aeb59ad92d877d2948cfe550d8e626803f864d5bb0e429a79dc6e0691a
Oct 31 00:03:10.355308 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:10.355271    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744" event={"ID":"dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9","Type":"ContainerStarted","Data":"568f93aeb59ad92d877d2948cfe550d8e626803f864d5bb0e429a79dc6e0691a"}
Oct 31 00:03:11.361515 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:11.361478    5078 generic.go:356] "Generic (PLEG): container finished" podID="dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9" containerID="9f810dc6d6f4d04b7e268ca852ff73b5df33b0dbb9ab90d55166f08579cf335e" exitCode=0
Oct 31 00:03:11.361948 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:11.361571    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744" event={"ID":"dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9","Type":"ContainerDied","Data":"9f810dc6d6f4d04b7e268ca852ff73b5df33b0dbb9ab90d55166f08579cf335e"}
Oct 31 00:03:12.502103 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:12.502079    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744"
Oct 31 00:03:12.660189 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:12.660097    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-maas-api-openshift-ca\") pod \"dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9\" (UID: \"dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9\") "
Oct 31 00:03:12.660189 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:12.660137    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-kube-api-access-9gt7n\" (UniqueName: \"kubernetes.io/projected/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-kube-api-access-9gt7n\") pod \"dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9\" (UID: \"dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9\") "
Oct 31 00:03:12.660189 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:12.660164    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-maas-api-xks-ca\") pod \"dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9\" (UID: \"dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9\") "
Oct 31 00:03:12.660516 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:12.660461    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-maas-api-openshift-ca" pod "dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9" (UID: "dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:03:12.662422 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:12.662399    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-maas-api-xks-ca" pod "dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9" (UID: "dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:03:12.662553 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:12.662536    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-kube-api-access-9gt7n" pod "dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9" (UID: "dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9"). InnerVolumeSpecName "kube-api-access-9gt7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:12.761301 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:12.761265    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:03:12.761301 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:12.761295    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:03:12.761301 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:12.761305    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gt7n\" (UniqueName: \"kubernetes.io/projected/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9-kube-api-access-9gt7n\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:03:13.371876 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:13.371846    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744"
Oct 31 00:03:13.372067 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:13.371884    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744" event={"ID":"dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9","Type":"ContainerDied","Data":"568f93aeb59ad92d877d2948cfe550d8e626803f864d5bb0e429a79dc6e0691a"}
Oct 31 00:03:13.372067 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:13.371936    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="568f93aeb59ad92d877d2948cfe550d8e626803f864d5bb0e429a79dc6e0691a"
Oct 31 00:03:17.590871 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:03:17.590834    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:03:33.222010 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.221974    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l"]
Oct 31 00:03:33.222440 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.222418    5078 cpu_manager.go:414] "containerMap: removing container" podUID="dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9" containerName="revoke-keys"
Oct 31 00:03:33.222440 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.222434    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9" containerName="revoke-keys"
Oct 31 00:03:33.222517 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.222489    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9" containerName="revoke-keys"
Oct 31 00:03:33.225497 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.225482    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l"
Oct 31 00:03:33.278174 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.278140    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:03:33.290155 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.290120    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-frd7l\" (UID: \"1ac1a712-bb7e-4680-8f22-20c1b19d38f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l"
Oct 31 00:03:33.290341 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.290244    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-frd7l\" (UID: \"1ac1a712-bb7e-4680-8f22-20c1b19d38f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l"
Oct 31 00:03:33.290341 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.290298    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kcvv\" (UniqueName: \"kubernetes.io/projected/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-kube-api-access-8kcvv\") pod \"maas-api-revoke-sub-0b7e8edb-frd7l\" (UID: \"1ac1a712-bb7e-4680-8f22-20c1b19d38f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l"
Oct 31 00:03:33.328062 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.328025    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:03:33.328062 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.328030    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:03:33.328270 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.328091    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:03:33.329842 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.329821    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l"]
Oct 31 00:03:33.391264 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.391227    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-frd7l\" (UID: \"1ac1a712-bb7e-4680-8f22-20c1b19d38f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l"
Oct 31 00:03:33.391464 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.391282    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-frd7l\" (UID: \"1ac1a712-bb7e-4680-8f22-20c1b19d38f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l"
Oct 31 00:03:33.391464 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.391316    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kcvv\" (UniqueName: \"kubernetes.io/projected/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-kube-api-access-8kcvv\") pod \"maas-api-revoke-sub-0b7e8edb-frd7l\" (UID: \"1ac1a712-bb7e-4680-8f22-20c1b19d38f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l"
Oct 31 00:03:33.391931 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.391907    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-frd7l\" (UID: \"1ac1a712-bb7e-4680-8f22-20c1b19d38f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l"
Oct 31 00:03:33.393900 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.393885    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-frd7l\" (UID: \"1ac1a712-bb7e-4680-8f22-20c1b19d38f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l"
Oct 31 00:03:33.448938 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.448909    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kcvv\" (UniqueName: \"kubernetes.io/projected/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-kube-api-access-8kcvv\") pod \"maas-api-revoke-sub-0b7e8edb-frd7l\" (UID: \"1ac1a712-bb7e-4680-8f22-20c1b19d38f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l"
Oct 31 00:03:33.538694 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.538615    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l"
Oct 31 00:03:33.910713 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:33.910647    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l"]
Oct 31 00:03:33.912956 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:03:33.912927    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac1a712_bb7e_4680_8f22_20c1b19d38f5.slice/crio-1c3525ff08221502851667a63051bfa37f3d826f4c94a4fb3aca0faef197f875 WatchSource:0}: Error finding container 1c3525ff08221502851667a63051bfa37f3d826f4c94a4fb3aca0faef197f875: Status 404 returned error can't find the container with id 1c3525ff08221502851667a63051bfa37f3d826f4c94a4fb3aca0faef197f875
Oct 31 00:03:34.494941 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:34.494901    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l" event={"ID":"1ac1a712-bb7e-4680-8f22-20c1b19d38f5","Type":"ContainerStarted","Data":"11780bf3f29ea26f0e5441c7043c9839bba75cd3116e3257f2f5e306f36896fc"}
Oct 31 00:03:34.495375 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:34.494971    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l" event={"ID":"1ac1a712-bb7e-4680-8f22-20c1b19d38f5","Type":"ContainerStarted","Data":"1c3525ff08221502851667a63051bfa37f3d826f4c94a4fb3aca0faef197f875"}
Oct 31 00:03:34.611957 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:34.611899    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l" podStartSLOduration=1.611884238 podStartE2EDuration="1.611884238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:34.610892176 +0000 UTC m=+3400.498798530" watchObservedRunningTime="2026-10-31 00:03:34.611884238 +0000 UTC m=+3400.499790591"
Oct 31 00:03:35.501120 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:35.501088    5078 generic.go:356] "Generic (PLEG): container finished" podID="1ac1a712-bb7e-4680-8f22-20c1b19d38f5" containerID="11780bf3f29ea26f0e5441c7043c9839bba75cd3116e3257f2f5e306f36896fc" exitCode=0
Oct 31 00:03:35.501533 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:35.501179    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l" event={"ID":"1ac1a712-bb7e-4680-8f22-20c1b19d38f5","Type":"ContainerDied","Data":"11780bf3f29ea26f0e5441c7043c9839bba75cd3116e3257f2f5e306f36896fc"}
Oct 31 00:03:36.636697 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:36.636673    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l"
Oct 31 00:03:36.722611 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:36.722573    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-kube-api-access-8kcvv\" (UniqueName: \"kubernetes.io/projected/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-kube-api-access-8kcvv\") pod \"1ac1a712-bb7e-4680-8f22-20c1b19d38f5\" (UID: \"1ac1a712-bb7e-4680-8f22-20c1b19d38f5\") "
Oct 31 00:03:36.722791 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:36.722701    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-maas-api-openshift-ca\") pod \"1ac1a712-bb7e-4680-8f22-20c1b19d38f5\" (UID: \"1ac1a712-bb7e-4680-8f22-20c1b19d38f5\") "
Oct 31 00:03:36.722791 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:36.722728    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-maas-api-xks-ca\") pod \"1ac1a712-bb7e-4680-8f22-20c1b19d38f5\" (UID: \"1ac1a712-bb7e-4680-8f22-20c1b19d38f5\") "
Oct 31 00:03:36.723082 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:36.723061    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-maas-api-openshift-ca" pod "1ac1a712-bb7e-4680-8f22-20c1b19d38f5" (UID: "1ac1a712-bb7e-4680-8f22-20c1b19d38f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:03:36.724948 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:36.724923    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-maas-api-xks-ca" pod "1ac1a712-bb7e-4680-8f22-20c1b19d38f5" (UID: "1ac1a712-bb7e-4680-8f22-20c1b19d38f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:03:36.725058 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:36.725040    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-kube-api-access-8kcvv" pod "1ac1a712-bb7e-4680-8f22-20c1b19d38f5" (UID: "1ac1a712-bb7e-4680-8f22-20c1b19d38f5"). InnerVolumeSpecName "kube-api-access-8kcvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:36.824217 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:36.824175    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:03:36.824217 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:36.824212    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:03:36.824217 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:36.824226    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kcvv\" (UniqueName: \"kubernetes.io/projected/1ac1a712-bb7e-4680-8f22-20c1b19d38f5-kube-api-access-8kcvv\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:03:37.512383 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:37.512352    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l"
Oct 31 00:03:37.512563 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:37.512356    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l" event={"ID":"1ac1a712-bb7e-4680-8f22-20c1b19d38f5","Type":"ContainerDied","Data":"1c3525ff08221502851667a63051bfa37f3d826f4c94a4fb3aca0faef197f875"}
Oct 31 00:03:37.512563 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:37.512460    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c3525ff08221502851667a63051bfa37f3d826f4c94a4fb3aca0faef197f875"
Oct 31 00:03:44.600201 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.600163    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g"]
Oct 31 00:03:44.600685 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.600671    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1ac1a712-bb7e-4680-8f22-20c1b19d38f5" containerName="revoke-keys"
Oct 31 00:03:44.600746 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.600686    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac1a712-bb7e-4680-8f22-20c1b19d38f5" containerName="revoke-keys"
Oct 31 00:03:44.600786 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.600754    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac1a712-bb7e-4680-8f22-20c1b19d38f5" containerName="revoke-keys"
Oct 31 00:03:44.603967 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.603947    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g"
Oct 31 00:03:44.688727 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.688689    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcffk\" (UniqueName: \"kubernetes.io/projected/0c822edc-daa5-4b1b-b830-d65e862ca425-kube-api-access-dcffk\") pod \"maas-api-revoke-sub-3d4156f1-7lx6g\" (UID: \"0c822edc-daa5-4b1b-b830-d65e862ca425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g"
Oct 31 00:03:44.688908 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.688747    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c822edc-daa5-4b1b-b830-d65e862ca425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7lx6g\" (UID: \"0c822edc-daa5-4b1b-b830-d65e862ca425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g"
Oct 31 00:03:44.688957 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.688912    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c822edc-daa5-4b1b-b830-d65e862ca425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7lx6g\" (UID: \"0c822edc-daa5-4b1b-b830-d65e862ca425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g"
Oct 31 00:03:44.707089 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.707061    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:03:44.707089 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.707076    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:03:44.707089 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.707066    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:03:44.707358 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.707158    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:03:44.709113 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.709089    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g"]
Oct 31 00:03:44.790402 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.790372    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c822edc-daa5-4b1b-b830-d65e862ca425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7lx6g\" (UID: \"0c822edc-daa5-4b1b-b830-d65e862ca425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g"
Oct 31 00:03:44.790581 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.790414    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcffk\" (UniqueName: \"kubernetes.io/projected/0c822edc-daa5-4b1b-b830-d65e862ca425-kube-api-access-dcffk\") pod \"maas-api-revoke-sub-3d4156f1-7lx6g\" (UID: \"0c822edc-daa5-4b1b-b830-d65e862ca425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g"
Oct 31 00:03:44.790581 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.790542    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c822edc-daa5-4b1b-b830-d65e862ca425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7lx6g\" (UID: \"0c822edc-daa5-4b1b-b830-d65e862ca425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g"
Oct 31 00:03:44.791098 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.791074    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c822edc-daa5-4b1b-b830-d65e862ca425-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7lx6g\" (UID: \"0c822edc-daa5-4b1b-b830-d65e862ca425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g"
Oct 31 00:03:44.793009 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.792985    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c822edc-daa5-4b1b-b830-d65e862ca425-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7lx6g\" (UID: \"0c822edc-daa5-4b1b-b830-d65e862ca425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g"
Oct 31 00:03:44.848618 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.848562    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcffk\" (UniqueName: \"kubernetes.io/projected/0c822edc-daa5-4b1b-b830-d65e862ca425-kube-api-access-dcffk\") pod \"maas-api-revoke-sub-3d4156f1-7lx6g\" (UID: \"0c822edc-daa5-4b1b-b830-d65e862ca425\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g"
Oct 31 00:03:44.916877 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:44.916822    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g"
Oct 31 00:03:45.286482 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:45.286458    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g"]
Oct 31 00:03:45.288529 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:03:45.288501    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c822edc_daa5_4b1b_b830_d65e862ca425.slice/crio-995356b4885b3e53243a920758a01b6a1e42bc02afc2f5cacce705bdcf7fbe33 WatchSource:0}: Error finding container 995356b4885b3e53243a920758a01b6a1e42bc02afc2f5cacce705bdcf7fbe33: Status 404 returned error can't find the container with id 995356b4885b3e53243a920758a01b6a1e42bc02afc2f5cacce705bdcf7fbe33
Oct 31 00:03:45.552344 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:45.552311    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g" event={"ID":"0c822edc-daa5-4b1b-b830-d65e862ca425","Type":"ContainerStarted","Data":"0ecd2b22ff0e147120bc10ecb15afd7de3116aa26be6ee576b1f8e4e619dc58d"}
Oct 31 00:03:45.552483 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:45.552363    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g" event={"ID":"0c822edc-daa5-4b1b-b830-d65e862ca425","Type":"ContainerStarted","Data":"995356b4885b3e53243a920758a01b6a1e42bc02afc2f5cacce705bdcf7fbe33"}
Oct 31 00:03:45.673460 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:45.673405    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g" podStartSLOduration=1.673389698 podStartE2EDuration="1.673389698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:45.672035594 +0000 UTC m=+3411.559941948" watchObservedRunningTime="2026-10-31 00:03:45.673389698 +0000 UTC m=+3411.561296051"
Oct 31 00:03:46.560804 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:46.560772    5078 generic.go:356] "Generic (PLEG): container finished" podID="0c822edc-daa5-4b1b-b830-d65e862ca425" containerID="0ecd2b22ff0e147120bc10ecb15afd7de3116aa26be6ee576b1f8e4e619dc58d" exitCode=0
Oct 31 00:03:46.560993 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:46.560841    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g" event={"ID":"0c822edc-daa5-4b1b-b830-d65e862ca425","Type":"ContainerDied","Data":"0ecd2b22ff0e147120bc10ecb15afd7de3116aa26be6ee576b1f8e4e619dc58d"}
Oct 31 00:03:47.727622 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:03:47.727173    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:03:48.277152 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:48.276859    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g"
Oct 31 00:03:48.368142 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:48.368107    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c822edc-daa5-4b1b-b830-d65e862ca425-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c822edc-daa5-4b1b-b830-d65e862ca425-maas-api-xks-ca\") pod \"0c822edc-daa5-4b1b-b830-d65e862ca425\" (UID: \"0c822edc-daa5-4b1b-b830-d65e862ca425\") "
Oct 31 00:03:48.368908 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:48.368881    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c822edc-daa5-4b1b-b830-d65e862ca425-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c822edc-daa5-4b1b-b830-d65e862ca425-maas-api-openshift-ca\") pod \"0c822edc-daa5-4b1b-b830-d65e862ca425\" (UID: \"0c822edc-daa5-4b1b-b830-d65e862ca425\") "
Oct 31 00:03:48.369120 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:48.369101    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c822edc-daa5-4b1b-b830-d65e862ca425-kube-api-access-dcffk\" (UniqueName: \"kubernetes.io/projected/0c822edc-daa5-4b1b-b830-d65e862ca425-kube-api-access-dcffk\") pod \"0c822edc-daa5-4b1b-b830-d65e862ca425\" (UID: \"0c822edc-daa5-4b1b-b830-d65e862ca425\") "
Oct 31 00:03:48.386092 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:48.386070    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c822edc-daa5-4b1b-b830-d65e862ca425-maas-api-openshift-ca" pod "0c822edc-daa5-4b1b-b830-d65e862ca425" (UID: "0c822edc-daa5-4b1b-b830-d65e862ca425"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:03:48.388184 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:48.388137    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c822edc-daa5-4b1b-b830-d65e862ca425-maas-api-xks-ca" pod "0c822edc-daa5-4b1b-b830-d65e862ca425" (UID: "0c822edc-daa5-4b1b-b830-d65e862ca425"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:03:48.394053 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:48.394014    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c822edc-daa5-4b1b-b830-d65e862ca425-kube-api-access-dcffk" pod "0c822edc-daa5-4b1b-b830-d65e862ca425" (UID: "0c822edc-daa5-4b1b-b830-d65e862ca425"). InnerVolumeSpecName "kube-api-access-dcffk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:03:48.490027 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:48.489693    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcffk\" (UniqueName: \"kubernetes.io/projected/0c822edc-daa5-4b1b-b830-d65e862ca425-kube-api-access-dcffk\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:03:48.490027 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:48.489728    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c822edc-daa5-4b1b-b830-d65e862ca425-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:03:48.490027 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:48.489745    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c822edc-daa5-4b1b-b830-d65e862ca425-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:03:48.592427 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:48.592393    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g" event={"ID":"0c822edc-daa5-4b1b-b830-d65e862ca425","Type":"ContainerDied","Data":"995356b4885b3e53243a920758a01b6a1e42bc02afc2f5cacce705bdcf7fbe33"}
Oct 31 00:03:48.592619 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:48.592432    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g"
Oct 31 00:03:48.592619 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:48.592444    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="995356b4885b3e53243a920758a01b6a1e42bc02afc2f5cacce705bdcf7fbe33"
Oct 31 00:03:59.054822 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:59.054788    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc"]
Oct 31 00:03:59.104330 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:03:59.104298    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-lcffc"]
Oct 31 00:04:00.299349 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:00.299308    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce3998c5-3854-4ef9-805f-41dace394532" path="/var/lib/kubelet/pods/ce3998c5-3854-4ef9-805f-41dace394532/volumes"
Oct 31 00:04:04.157392 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:04.157318    5078 scope.go:122] "RemoveContainer" containerID="d2398f29837f450735e03a1d063915728317efe264a6178af705350fe5e9bd55"
Oct 31 00:04:04.165723 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:04.165701    5078 scope.go:122] "RemoveContainer" containerID="99c6f160c8f14951e06034d2dbde35bf9c7c20166ec33e48323588eaa9c07a2c"
Oct 31 00:04:04.173451 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:04.173418    5078 scope.go:122] "RemoveContainer" containerID="43112cb646ecf4361eb7a2fdad5a8bbfe505d192351b5fbde662b6d3ebf013ef"
Oct 31 00:04:04.180660 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:04.180637    5078 scope.go:122] "RemoveContainer" containerID="035ab9801fd6a064269477dcc0a84ed696c91acf32cbe710a754a0ae1dd154d8"
Oct 31 00:04:04.187533 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:04.187506    5078 scope.go:122] "RemoveContainer" containerID="39f46449da218ecb927d78c20ccedb2f10aca3b4875361ba8257e18930b2787c"
Oct 31 00:04:04.196845 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:04.196818    5078 scope.go:122] "RemoveContainer" containerID="cce43484a5d918bcd9064fdef73cc72d4824180579f36b3f25ba6984de35be1e"
Oct 31 00:04:05.675449 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.675419    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g"]
Oct 31 00:04:05.724907 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.724878    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7lx6g"]
Oct 31 00:04:05.803829 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.803797    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp"]
Oct 31 00:04:05.804127 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.804115    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0c822edc-daa5-4b1b-b830-d65e862ca425" containerName="revoke-keys"
Oct 31 00:04:05.804174 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.804128    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c822edc-daa5-4b1b-b830-d65e862ca425" containerName="revoke-keys"
Oct 31 00:04:05.804211 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.804193    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c822edc-daa5-4b1b-b830-d65e862ca425" containerName="revoke-keys"
Oct 31 00:04:05.809064 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.809038    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp"
Oct 31 00:04:05.864888 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.864847    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp"]
Oct 31 00:04:05.867806 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.867784    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:04:05.867806 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.867793    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:04:05.868217 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.868200    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:04:05.891722 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.891684    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqqxq\" (UniqueName: \"kubernetes.io/projected/3a122838-2da8-4514-8ad5-9134fc68c530-kube-api-access-lqqxq\") pod \"maas-api-revoke-sub-3d4156f1-mf4rp\" (UID: \"3a122838-2da8-4514-8ad5-9134fc68c530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp"
Oct 31 00:04:05.891879 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.891727    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a122838-2da8-4514-8ad5-9134fc68c530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mf4rp\" (UID: \"3a122838-2da8-4514-8ad5-9134fc68c530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp"
Oct 31 00:04:05.891879 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.891809    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a122838-2da8-4514-8ad5-9134fc68c530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mf4rp\" (UID: \"3a122838-2da8-4514-8ad5-9134fc68c530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp"
Oct 31 00:04:05.912869 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.912850    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:04:05.992859 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.992797    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqqxq\" (UniqueName: \"kubernetes.io/projected/3a122838-2da8-4514-8ad5-9134fc68c530-kube-api-access-lqqxq\") pod \"maas-api-revoke-sub-3d4156f1-mf4rp\" (UID: \"3a122838-2da8-4514-8ad5-9134fc68c530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp"
Oct 31 00:04:05.992859 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.992834    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a122838-2da8-4514-8ad5-9134fc68c530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mf4rp\" (UID: \"3a122838-2da8-4514-8ad5-9134fc68c530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp"
Oct 31 00:04:05.993037 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.992964    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a122838-2da8-4514-8ad5-9134fc68c530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mf4rp\" (UID: \"3a122838-2da8-4514-8ad5-9134fc68c530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp"
Oct 31 00:04:05.993563 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.993540    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a122838-2da8-4514-8ad5-9134fc68c530-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-mf4rp\" (UID: \"3a122838-2da8-4514-8ad5-9134fc68c530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp"
Oct 31 00:04:05.995250 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:05.995233    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a122838-2da8-4514-8ad5-9134fc68c530-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-mf4rp\" (UID: \"3a122838-2da8-4514-8ad5-9134fc68c530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp"
Oct 31 00:04:06.051666 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:06.051645    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqqxq\" (UniqueName: \"kubernetes.io/projected/3a122838-2da8-4514-8ad5-9134fc68c530-kube-api-access-lqqxq\") pod \"maas-api-revoke-sub-3d4156f1-mf4rp\" (UID: \"3a122838-2da8-4514-8ad5-9134fc68c530\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp"
Oct 31 00:04:06.121272 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:06.121244    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp"
Oct 31 00:04:06.296067 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:06.296034    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c822edc-daa5-4b1b-b830-d65e862ca425" path="/var/lib/kubelet/pods/0c822edc-daa5-4b1b-b830-d65e862ca425/volumes"
Oct 31 00:04:06.494188 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:06.494159    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp"]
Oct 31 00:04:06.494732 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:04:06.494702    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a122838_2da8_4514_8ad5_9134fc68c530.slice/crio-523cc3665ac4ad7c7c1c845687d6345215b23bff8f6278b27d2e872afb966743 WatchSource:0}: Error finding container 523cc3665ac4ad7c7c1c845687d6345215b23bff8f6278b27d2e872afb966743: Status 404 returned error can't find the container with id 523cc3665ac4ad7c7c1c845687d6345215b23bff8f6278b27d2e872afb966743
Oct 31 00:04:06.723665 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:06.723627    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp" event={"ID":"3a122838-2da8-4514-8ad5-9134fc68c530","Type":"ContainerStarted","Data":"523cc3665ac4ad7c7c1c845687d6345215b23bff8f6278b27d2e872afb966743"}
Oct 31 00:04:07.730065 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:07.729986    5078 generic.go:356] "Generic (PLEG): container finished" podID="3a122838-2da8-4514-8ad5-9134fc68c530" containerID="88232ab8b3bf06ffc9bc542d0c5b5f8058503f62e0e0bcb86bf6ba29fc127ad3" exitCode=0
Oct 31 00:04:07.730065 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:07.730033    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp" event={"ID":"3a122838-2da8-4514-8ad5-9134fc68c530","Type":"ContainerDied","Data":"88232ab8b3bf06ffc9bc542d0c5b5f8058503f62e0e0bcb86bf6ba29fc127ad3"}
Oct 31 00:04:08.860496 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:08.860474    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp"
Oct 31 00:04:08.921253 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:08.921224    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a122838-2da8-4514-8ad5-9134fc68c530-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a122838-2da8-4514-8ad5-9134fc68c530-maas-api-openshift-ca\") pod \"3a122838-2da8-4514-8ad5-9134fc68c530\" (UID: \"3a122838-2da8-4514-8ad5-9134fc68c530\") "
Oct 31 00:04:08.921374 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:08.921285    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a122838-2da8-4514-8ad5-9134fc68c530-kube-api-access-lqqxq\" (UniqueName: \"kubernetes.io/projected/3a122838-2da8-4514-8ad5-9134fc68c530-kube-api-access-lqqxq\") pod \"3a122838-2da8-4514-8ad5-9134fc68c530\" (UID: \"3a122838-2da8-4514-8ad5-9134fc68c530\") "
Oct 31 00:04:08.921374 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:08.921340    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a122838-2da8-4514-8ad5-9134fc68c530-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a122838-2da8-4514-8ad5-9134fc68c530-maas-api-xks-ca\") pod \"3a122838-2da8-4514-8ad5-9134fc68c530\" (UID: \"3a122838-2da8-4514-8ad5-9134fc68c530\") "
Oct 31 00:04:08.921580 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:08.921556    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a122838-2da8-4514-8ad5-9134fc68c530-maas-api-openshift-ca" pod "3a122838-2da8-4514-8ad5-9134fc68c530" (UID: "3a122838-2da8-4514-8ad5-9134fc68c530"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:04:08.923404 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:08.923384    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a122838-2da8-4514-8ad5-9134fc68c530-kube-api-access-lqqxq" pod "3a122838-2da8-4514-8ad5-9134fc68c530" (UID: "3a122838-2da8-4514-8ad5-9134fc68c530"). InnerVolumeSpecName "kube-api-access-lqqxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:04:08.923467 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:08.923382    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a122838-2da8-4514-8ad5-9134fc68c530-maas-api-xks-ca" pod "3a122838-2da8-4514-8ad5-9134fc68c530" (UID: "3a122838-2da8-4514-8ad5-9134fc68c530"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:04:09.022197 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:09.022128    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqqxq\" (UniqueName: \"kubernetes.io/projected/3a122838-2da8-4514-8ad5-9134fc68c530-kube-api-access-lqqxq\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:09.022197 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:09.022155    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a122838-2da8-4514-8ad5-9134fc68c530-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:09.022197 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:09.022166    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a122838-2da8-4514-8ad5-9134fc68c530-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:09.739959 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:09.739930    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp"
Oct 31 00:04:09.740150 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:09.739966    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp" event={"ID":"3a122838-2da8-4514-8ad5-9134fc68c530","Type":"ContainerDied","Data":"523cc3665ac4ad7c7c1c845687d6345215b23bff8f6278b27d2e872afb966743"}
Oct 31 00:04:09.740150 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:09.740011    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="523cc3665ac4ad7c7c1c845687d6345215b23bff8f6278b27d2e872afb966743"
Oct 31 00:04:17.590804 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:04:17.590770    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:04:40.407798 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.407766    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj"]
Oct 31 00:04:40.408267 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.408109    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3a122838-2da8-4514-8ad5-9134fc68c530" containerName="revoke-keys"
Oct 31 00:04:40.408267 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.408120    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a122838-2da8-4514-8ad5-9134fc68c530" containerName="revoke-keys"
Oct 31 00:04:40.408267 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.408185    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a122838-2da8-4514-8ad5-9134fc68c530" containerName="revoke-keys"
Oct 31 00:04:40.411259 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.411243    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj"
Oct 31 00:04:40.513842 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.513805    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:04:40.514032 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.513887    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:04:40.514032 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.513887    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:04:40.514032 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.513964    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:04:40.515712 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.515695    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj"]
Oct 31 00:04:40.551568 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.551524    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xt42\" (UniqueName: \"kubernetes.io/projected/9941253b-db94-4b2e-ba32-1d1034bb8a57-kube-api-access-7xt42\") pod \"maas-api-revoke-sub-e2fc9c5b-qndcj\" (UID: \"9941253b-db94-4b2e-ba32-1d1034bb8a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj"
Oct 31 00:04:40.551748 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.551609    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9941253b-db94-4b2e-ba32-1d1034bb8a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qndcj\" (UID: \"9941253b-db94-4b2e-ba32-1d1034bb8a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj"
Oct 31 00:04:40.551828 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.551756    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9941253b-db94-4b2e-ba32-1d1034bb8a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qndcj\" (UID: \"9941253b-db94-4b2e-ba32-1d1034bb8a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj"
Oct 31 00:04:40.652660 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.652615    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9941253b-db94-4b2e-ba32-1d1034bb8a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qndcj\" (UID: \"9941253b-db94-4b2e-ba32-1d1034bb8a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj"
Oct 31 00:04:40.652824 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.652698    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9941253b-db94-4b2e-ba32-1d1034bb8a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qndcj\" (UID: \"9941253b-db94-4b2e-ba32-1d1034bb8a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj"
Oct 31 00:04:40.652824 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.652738    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xt42\" (UniqueName: \"kubernetes.io/projected/9941253b-db94-4b2e-ba32-1d1034bb8a57-kube-api-access-7xt42\") pod \"maas-api-revoke-sub-e2fc9c5b-qndcj\" (UID: \"9941253b-db94-4b2e-ba32-1d1034bb8a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj"
Oct 31 00:04:40.653382 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.653297    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9941253b-db94-4b2e-ba32-1d1034bb8a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qndcj\" (UID: \"9941253b-db94-4b2e-ba32-1d1034bb8a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj"
Oct 31 00:04:40.655032 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.655012    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9941253b-db94-4b2e-ba32-1d1034bb8a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qndcj\" (UID: \"9941253b-db94-4b2e-ba32-1d1034bb8a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj"
Oct 31 00:04:40.711255 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.711197    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xt42\" (UniqueName: \"kubernetes.io/projected/9941253b-db94-4b2e-ba32-1d1034bb8a57-kube-api-access-7xt42\") pod \"maas-api-revoke-sub-e2fc9c5b-qndcj\" (UID: \"9941253b-db94-4b2e-ba32-1d1034bb8a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj"
Oct 31 00:04:40.728160 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:40.728137    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj"
Oct 31 00:04:41.115533 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:41.115509    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj"]
Oct 31 00:04:41.118075 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:04:41.118041    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9941253b_db94_4b2e_ba32_1d1034bb8a57.slice/crio-e38792d739f38fc5cf9fa37ef8fa20392e64141b9fa6c2484b02620f8cce8557 WatchSource:0}: Error finding container e38792d739f38fc5cf9fa37ef8fa20392e64141b9fa6c2484b02620f8cce8557: Status 404 returned error can't find the container with id e38792d739f38fc5cf9fa37ef8fa20392e64141b9fa6c2484b02620f8cce8557
Oct 31 00:04:41.932796 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:41.932758    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj" event={"ID":"9941253b-db94-4b2e-ba32-1d1034bb8a57","Type":"ContainerStarted","Data":"0bb61604f18dc737641e16f46de0a30c6edd03f108ed4ac17dc3121f56733bf8"}
Oct 31 00:04:41.933242 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:41.932854    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj" event={"ID":"9941253b-db94-4b2e-ba32-1d1034bb8a57","Type":"ContainerStarted","Data":"e38792d739f38fc5cf9fa37ef8fa20392e64141b9fa6c2484b02620f8cce8557"}
Oct 31 00:04:42.099858 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:42.099800    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj" podStartSLOduration=2.099784588 podStartE2EDuration="2.099784588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:42.049037911 +0000 UTC m=+3467.936944257" watchObservedRunningTime="2026-10-31 00:04:42.099784588 +0000 UTC m=+3467.987690978"
Oct 31 00:04:42.938744 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:42.938705    5078 generic.go:356] "Generic (PLEG): container finished" podID="9941253b-db94-4b2e-ba32-1d1034bb8a57" containerID="0bb61604f18dc737641e16f46de0a30c6edd03f108ed4ac17dc3121f56733bf8" exitCode=0
Oct 31 00:04:42.939149 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:42.938800    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj" event={"ID":"9941253b-db94-4b2e-ba32-1d1034bb8a57","Type":"ContainerDied","Data":"0bb61604f18dc737641e16f46de0a30c6edd03f108ed4ac17dc3121f56733bf8"}
Oct 31 00:04:44.079612 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:44.079580    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj"
Oct 31 00:04:44.186038 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:44.186009    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9941253b-db94-4b2e-ba32-1d1034bb8a57-kube-api-access-7xt42\" (UniqueName: \"kubernetes.io/projected/9941253b-db94-4b2e-ba32-1d1034bb8a57-kube-api-access-7xt42\") pod \"9941253b-db94-4b2e-ba32-1d1034bb8a57\" (UID: \"9941253b-db94-4b2e-ba32-1d1034bb8a57\") "
Oct 31 00:04:44.186175 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:44.186108    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9941253b-db94-4b2e-ba32-1d1034bb8a57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9941253b-db94-4b2e-ba32-1d1034bb8a57-maas-api-openshift-ca\") pod \"9941253b-db94-4b2e-ba32-1d1034bb8a57\" (UID: \"9941253b-db94-4b2e-ba32-1d1034bb8a57\") "
Oct 31 00:04:44.186175 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:44.186134    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9941253b-db94-4b2e-ba32-1d1034bb8a57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9941253b-db94-4b2e-ba32-1d1034bb8a57-maas-api-xks-ca\") pod \"9941253b-db94-4b2e-ba32-1d1034bb8a57\" (UID: \"9941253b-db94-4b2e-ba32-1d1034bb8a57\") "
Oct 31 00:04:44.186451 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:44.186423    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9941253b-db94-4b2e-ba32-1d1034bb8a57-maas-api-openshift-ca" pod "9941253b-db94-4b2e-ba32-1d1034bb8a57" (UID: "9941253b-db94-4b2e-ba32-1d1034bb8a57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:04:44.188212 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:44.188191    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9941253b-db94-4b2e-ba32-1d1034bb8a57-kube-api-access-7xt42" pod "9941253b-db94-4b2e-ba32-1d1034bb8a57" (UID: "9941253b-db94-4b2e-ba32-1d1034bb8a57"). InnerVolumeSpecName "kube-api-access-7xt42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:04:44.188301 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:44.188281    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9941253b-db94-4b2e-ba32-1d1034bb8a57-maas-api-xks-ca" pod "9941253b-db94-4b2e-ba32-1d1034bb8a57" (UID: "9941253b-db94-4b2e-ba32-1d1034bb8a57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:04:44.287510 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:44.287429    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9941253b-db94-4b2e-ba32-1d1034bb8a57-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:44.287510 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:44.287457    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9941253b-db94-4b2e-ba32-1d1034bb8a57-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:44.287510 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:44.287468    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xt42\" (UniqueName: \"kubernetes.io/projected/9941253b-db94-4b2e-ba32-1d1034bb8a57-kube-api-access-7xt42\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:44.950030 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:44.950004    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj"
Oct 31 00:04:44.950030 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:44.950016    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj" event={"ID":"9941253b-db94-4b2e-ba32-1d1034bb8a57","Type":"ContainerDied","Data":"e38792d739f38fc5cf9fa37ef8fa20392e64141b9fa6c2484b02620f8cce8557"}
Oct 31 00:04:44.950265 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:44.950061    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e38792d739f38fc5cf9fa37ef8fa20392e64141b9fa6c2484b02620f8cce8557"
Oct 31 00:04:47.589946 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:04:47.589905    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:04:51.231839 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.231810    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf"]
Oct 31 00:04:51.232234 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.232140    5078 cpu_manager.go:414] "containerMap: removing container" podUID="9941253b-db94-4b2e-ba32-1d1034bb8a57" containerName="revoke-keys"
Oct 31 00:04:51.232234 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.232151    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9941253b-db94-4b2e-ba32-1d1034bb8a57" containerName="revoke-keys"
Oct 31 00:04:51.232234 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.232217    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="9941253b-db94-4b2e-ba32-1d1034bb8a57" containerName="revoke-keys"
Oct 31 00:04:51.235293 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.235276    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf"
Oct 31 00:04:51.290250 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.290224    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:04:51.337846 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.337818    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:04:51.338030 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.337898    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:04:51.338030 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.337915    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:04:51.340051 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.340032    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf"]
Oct 31 00:04:51.351363 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.351344    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ff93983-c7ab-4e70-a4c0-a43d66390539-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-9fvrf\" (UID: \"5ff93983-c7ab-4e70-a4c0-a43d66390539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf"
Oct 31 00:04:51.351433 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.351390    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8pr8\" (UniqueName: \"kubernetes.io/projected/5ff93983-c7ab-4e70-a4c0-a43d66390539-kube-api-access-p8pr8\") pod \"maas-api-revoke-sub-5a29efb3-9fvrf\" (UID: \"5ff93983-c7ab-4e70-a4c0-a43d66390539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf"
Oct 31 00:04:51.351499 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.351482    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ff93983-c7ab-4e70-a4c0-a43d66390539-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-9fvrf\" (UID: \"5ff93983-c7ab-4e70-a4c0-a43d66390539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf"
Oct 31 00:04:51.452704 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.452677    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ff93983-c7ab-4e70-a4c0-a43d66390539-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-9fvrf\" (UID: \"5ff93983-c7ab-4e70-a4c0-a43d66390539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf"
Oct 31 00:04:51.452890 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.452734    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8pr8\" (UniqueName: \"kubernetes.io/projected/5ff93983-c7ab-4e70-a4c0-a43d66390539-kube-api-access-p8pr8\") pod \"maas-api-revoke-sub-5a29efb3-9fvrf\" (UID: \"5ff93983-c7ab-4e70-a4c0-a43d66390539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf"
Oct 31 00:04:51.452890 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.452774    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ff93983-c7ab-4e70-a4c0-a43d66390539-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-9fvrf\" (UID: \"5ff93983-c7ab-4e70-a4c0-a43d66390539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf"
Oct 31 00:04:51.453393 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.453374    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ff93983-c7ab-4e70-a4c0-a43d66390539-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-9fvrf\" (UID: \"5ff93983-c7ab-4e70-a4c0-a43d66390539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf"
Oct 31 00:04:51.455188 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.455167    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ff93983-c7ab-4e70-a4c0-a43d66390539-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-9fvrf\" (UID: \"5ff93983-c7ab-4e70-a4c0-a43d66390539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf"
Oct 31 00:04:51.512377 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.512326    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8pr8\" (UniqueName: \"kubernetes.io/projected/5ff93983-c7ab-4e70-a4c0-a43d66390539-kube-api-access-p8pr8\") pod \"maas-api-revoke-sub-5a29efb3-9fvrf\" (UID: \"5ff93983-c7ab-4e70-a4c0-a43d66390539\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf"
Oct 31 00:04:51.552201 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.552181    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf"
Oct 31 00:04:51.925724 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.925700    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf"]
Oct 31 00:04:51.926974 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:04:51.926944    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff93983_c7ab_4e70_a4c0_a43d66390539.slice/crio-a35e8ebc1fb6159c6785110265b860d4ac976f151e0f72a8fb33acfd82edc2a9 WatchSource:0}: Error finding container a35e8ebc1fb6159c6785110265b860d4ac976f151e0f72a8fb33acfd82edc2a9: Status 404 returned error can't find the container with id a35e8ebc1fb6159c6785110265b860d4ac976f151e0f72a8fb33acfd82edc2a9
Oct 31 00:04:51.984981 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:51.984954    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf" event={"ID":"5ff93983-c7ab-4e70-a4c0-a43d66390539","Type":"ContainerStarted","Data":"a35e8ebc1fb6159c6785110265b860d4ac976f151e0f72a8fb33acfd82edc2a9"}
Oct 31 00:04:52.998252 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:52.998218    5078 generic.go:356] "Generic (PLEG): container finished" podID="5ff93983-c7ab-4e70-a4c0-a43d66390539" containerID="84eaf4fad5cf36fb8a403b86db93933ea09ce59fa69e57cb6b3955b203fe341d" exitCode=0
Oct 31 00:04:52.998647 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:52.998264    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf" event={"ID":"5ff93983-c7ab-4e70-a4c0-a43d66390539","Type":"ContainerDied","Data":"84eaf4fad5cf36fb8a403b86db93933ea09ce59fa69e57cb6b3955b203fe341d"}
Oct 31 00:04:54.183689 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:54.183655    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf"
Oct 31 00:04:54.277493 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:54.277387    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ff93983-c7ab-4e70-a4c0-a43d66390539-kube-api-access-p8pr8\" (UniqueName: \"kubernetes.io/projected/5ff93983-c7ab-4e70-a4c0-a43d66390539-kube-api-access-p8pr8\") pod \"5ff93983-c7ab-4e70-a4c0-a43d66390539\" (UID: \"5ff93983-c7ab-4e70-a4c0-a43d66390539\") "
Oct 31 00:04:54.277493 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:54.277451    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ff93983-c7ab-4e70-a4c0-a43d66390539-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ff93983-c7ab-4e70-a4c0-a43d66390539-maas-api-openshift-ca\") pod \"5ff93983-c7ab-4e70-a4c0-a43d66390539\" (UID: \"5ff93983-c7ab-4e70-a4c0-a43d66390539\") "
Oct 31 00:04:54.277788 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:54.277585    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ff93983-c7ab-4e70-a4c0-a43d66390539-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ff93983-c7ab-4e70-a4c0-a43d66390539-maas-api-xks-ca\") pod \"5ff93983-c7ab-4e70-a4c0-a43d66390539\" (UID: \"5ff93983-c7ab-4e70-a4c0-a43d66390539\") "
Oct 31 00:04:54.278054 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:54.278005    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ff93983-c7ab-4e70-a4c0-a43d66390539-maas-api-openshift-ca" pod "5ff93983-c7ab-4e70-a4c0-a43d66390539" (UID: "5ff93983-c7ab-4e70-a4c0-a43d66390539"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:04:54.280380 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:54.280303    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ff93983-c7ab-4e70-a4c0-a43d66390539-kube-api-access-p8pr8" pod "5ff93983-c7ab-4e70-a4c0-a43d66390539" (UID: "5ff93983-c7ab-4e70-a4c0-a43d66390539"). InnerVolumeSpecName "kube-api-access-p8pr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:04:54.280964 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:54.280939    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ff93983-c7ab-4e70-a4c0-a43d66390539-maas-api-xks-ca" pod "5ff93983-c7ab-4e70-a4c0-a43d66390539" (UID: "5ff93983-c7ab-4e70-a4c0-a43d66390539"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:04:54.379321 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:54.379237    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ff93983-c7ab-4e70-a4c0-a43d66390539-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:54.379321 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:54.379277    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8pr8\" (UniqueName: \"kubernetes.io/projected/5ff93983-c7ab-4e70-a4c0-a43d66390539-kube-api-access-p8pr8\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:54.379321 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:54.379295    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ff93983-c7ab-4e70-a4c0-a43d66390539-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:55.029245 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:55.029203    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf" event={"ID":"5ff93983-c7ab-4e70-a4c0-a43d66390539","Type":"ContainerDied","Data":"a35e8ebc1fb6159c6785110265b860d4ac976f151e0f72a8fb33acfd82edc2a9"}
Oct 31 00:04:55.029734 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:55.029659    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a35e8ebc1fb6159c6785110265b860d4ac976f151e0f72a8fb33acfd82edc2a9"
Oct 31 00:04:55.029914 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:04:55.029561    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf"
Oct 31 00:05:04.230033 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:04.230002    5078 scope.go:122] "RemoveContainer" containerID="6dce4d7891d94716744db6a2dee30f5b6671a1b377943c4c4c069ad06e6f5246"
Oct 31 00:05:07.923121 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:07.923077    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp"]
Oct 31 00:05:07.972825 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:07.972791    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-mf4rp"]
Oct 31 00:05:08.061310 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.061279    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2"]
Oct 31 00:05:08.061665 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.061652    5078 cpu_manager.go:414] "containerMap: removing container" podUID="5ff93983-c7ab-4e70-a4c0-a43d66390539" containerName="revoke-keys"
Oct 31 00:05:08.061732 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.061666    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ff93983-c7ab-4e70-a4c0-a43d66390539" containerName="revoke-keys"
Oct 31 00:05:08.061732 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.061730    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ff93983-c7ab-4e70-a4c0-a43d66390539" containerName="revoke-keys"
Oct 31 00:05:08.065845 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.065830    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2"
Oct 31 00:05:08.120895 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.120864    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:05:08.121468 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.121447    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:05:08.121541 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.121475    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:05:08.121541 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.121513    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:05:08.127277 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.127243    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2"]
Oct 31 00:05:08.147317 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.147285    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2lvz\" (UniqueName: \"kubernetes.io/projected/1fad9310-270a-46ba-853f-02c682398111-kube-api-access-r2lvz\") pod \"maas-api-revoke-sub-3d4156f1-7kcg2\" (UID: \"1fad9310-270a-46ba-853f-02c682398111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2"
Oct 31 00:05:08.147481 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.147350    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fad9310-270a-46ba-853f-02c682398111-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7kcg2\" (UID: \"1fad9310-270a-46ba-853f-02c682398111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2"
Oct 31 00:05:08.147530 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.147489    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fad9310-270a-46ba-853f-02c682398111-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7kcg2\" (UID: \"1fad9310-270a-46ba-853f-02c682398111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2"
Oct 31 00:05:08.249041 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.248960    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2lvz\" (UniqueName: \"kubernetes.io/projected/1fad9310-270a-46ba-853f-02c682398111-kube-api-access-r2lvz\") pod \"maas-api-revoke-sub-3d4156f1-7kcg2\" (UID: \"1fad9310-270a-46ba-853f-02c682398111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2"
Oct 31 00:05:08.249041 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.249036    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fad9310-270a-46ba-853f-02c682398111-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7kcg2\" (UID: \"1fad9310-270a-46ba-853f-02c682398111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2"
Oct 31 00:05:08.249244 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.249070    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fad9310-270a-46ba-853f-02c682398111-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7kcg2\" (UID: \"1fad9310-270a-46ba-853f-02c682398111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2"
Oct 31 00:05:08.249862 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.249835    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fad9310-270a-46ba-853f-02c682398111-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-7kcg2\" (UID: \"1fad9310-270a-46ba-853f-02c682398111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2"
Oct 31 00:05:08.251611 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.251577    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fad9310-270a-46ba-853f-02c682398111-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-7kcg2\" (UID: \"1fad9310-270a-46ba-853f-02c682398111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2"
Oct 31 00:05:08.296012 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.295975    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a122838-2da8-4514-8ad5-9134fc68c530" path="/var/lib/kubelet/pods/3a122838-2da8-4514-8ad5-9134fc68c530/volumes"
Oct 31 00:05:08.307453 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.307430    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2lvz\" (UniqueName: \"kubernetes.io/projected/1fad9310-270a-46ba-853f-02c682398111-kube-api-access-r2lvz\") pod \"maas-api-revoke-sub-3d4156f1-7kcg2\" (UID: \"1fad9310-270a-46ba-853f-02c682398111\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2"
Oct 31 00:05:08.378176 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.378138    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2"
Oct 31 00:05:08.754224 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:08.754199    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2"]
Oct 31 00:05:08.756215 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:05:08.756183    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fad9310_270a_46ba_853f_02c682398111.slice/crio-c7ca28e66f3936e8294e8341bd84fe900cb44bd6ce5ad55418c895804ccba513 WatchSource:0}: Error finding container c7ca28e66f3936e8294e8341bd84fe900cb44bd6ce5ad55418c895804ccba513: Status 404 returned error can't find the container with id c7ca28e66f3936e8294e8341bd84fe900cb44bd6ce5ad55418c895804ccba513
Oct 31 00:05:09.099443 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:09.099406    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2" event={"ID":"1fad9310-270a-46ba-853f-02c682398111","Type":"ContainerStarted","Data":"4331cacfc410be11248002cf55c817647efb7c3fa3ddb61d56f4f08e5f28b59e"}
Oct 31 00:05:09.099868 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:09.099464    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2" event={"ID":"1fad9310-270a-46ba-853f-02c682398111","Type":"ContainerStarted","Data":"c7ca28e66f3936e8294e8341bd84fe900cb44bd6ce5ad55418c895804ccba513"}
Oct 31 00:05:09.222357 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:09.222304    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2" podStartSLOduration=2.222288127 podStartE2EDuration="2.222288127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:05:09.221661277 +0000 UTC m=+3495.109567631" watchObservedRunningTime="2026-10-31 00:05:09.222288127 +0000 UTC m=+3495.110194479"
Oct 31 00:05:10.105564 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:10.105530    5078 generic.go:356] "Generic (PLEG): container finished" podID="1fad9310-270a-46ba-853f-02c682398111" containerID="4331cacfc410be11248002cf55c817647efb7c3fa3ddb61d56f4f08e5f28b59e" exitCode=0
Oct 31 00:05:10.106040 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:10.105575    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2" event={"ID":"1fad9310-270a-46ba-853f-02c682398111","Type":"ContainerDied","Data":"4331cacfc410be11248002cf55c817647efb7c3fa3ddb61d56f4f08e5f28b59e"}
Oct 31 00:05:11.245300 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:11.245277    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2"
Oct 31 00:05:11.277701 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:11.277676    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fad9310-270a-46ba-853f-02c682398111-kube-api-access-r2lvz\" (UniqueName: \"kubernetes.io/projected/1fad9310-270a-46ba-853f-02c682398111-kube-api-access-r2lvz\") pod \"1fad9310-270a-46ba-853f-02c682398111\" (UID: \"1fad9310-270a-46ba-853f-02c682398111\") "
Oct 31 00:05:11.277823 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:11.277759    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fad9310-270a-46ba-853f-02c682398111-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fad9310-270a-46ba-853f-02c682398111-maas-api-xks-ca\") pod \"1fad9310-270a-46ba-853f-02c682398111\" (UID: \"1fad9310-270a-46ba-853f-02c682398111\") "
Oct 31 00:05:11.277872 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:11.277843    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fad9310-270a-46ba-853f-02c682398111-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fad9310-270a-46ba-853f-02c682398111-maas-api-openshift-ca\") pod \"1fad9310-270a-46ba-853f-02c682398111\" (UID: \"1fad9310-270a-46ba-853f-02c682398111\") "
Oct 31 00:05:11.278208 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:11.278180    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fad9310-270a-46ba-853f-02c682398111-maas-api-openshift-ca" pod "1fad9310-270a-46ba-853f-02c682398111" (UID: "1fad9310-270a-46ba-853f-02c682398111"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:05:11.279792 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:11.279764    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fad9310-270a-46ba-853f-02c682398111-kube-api-access-r2lvz" pod "1fad9310-270a-46ba-853f-02c682398111" (UID: "1fad9310-270a-46ba-853f-02c682398111"). InnerVolumeSpecName "kube-api-access-r2lvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:05:11.279883 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:11.279796    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fad9310-270a-46ba-853f-02c682398111-maas-api-xks-ca" pod "1fad9310-270a-46ba-853f-02c682398111" (UID: "1fad9310-270a-46ba-853f-02c682398111"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:05:11.379740 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:11.379654    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2lvz\" (UniqueName: \"kubernetes.io/projected/1fad9310-270a-46ba-853f-02c682398111-kube-api-access-r2lvz\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:05:11.379740 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:11.379683    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fad9310-270a-46ba-853f-02c682398111-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:05:11.379740 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:11.379696    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fad9310-270a-46ba-853f-02c682398111-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:05:12.116012 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:12.115981    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2"
Oct 31 00:05:12.116187 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:12.116004    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2" event={"ID":"1fad9310-270a-46ba-853f-02c682398111","Type":"ContainerDied","Data":"c7ca28e66f3936e8294e8341bd84fe900cb44bd6ce5ad55418c895804ccba513"}
Oct 31 00:05:12.116187 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:12.116083    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7ca28e66f3936e8294e8341bd84fe900cb44bd6ce5ad55418c895804ccba513"
Oct 31 00:05:17.590008 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:05:17.589969    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:05:24.491549 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.491506    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"]
Oct 31 00:05:24.491985 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.491887    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1fad9310-270a-46ba-853f-02c682398111" containerName="revoke-keys"
Oct 31 00:05:24.491985 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.491903    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fad9310-270a-46ba-853f-02c682398111" containerName="revoke-keys"
Oct 31 00:05:24.492124 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.492109    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fad9310-270a-46ba-853f-02c682398111" containerName="revoke-keys"
Oct 31 00:05:24.495331 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.495314    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.548060 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.548019    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-9b1704-openshift-default-dockercfg-bvgfw\""
Oct 31 00:05:24.599640 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.599578    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"]
Oct 31 00:05:24.600043 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.600011    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmjwj\" (UniqueName: \"kubernetes.io/projected/fcae77a1-5934-42d2-8e88-d93e15356386-kube-api-access-mmjwj\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.600104 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.600086    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcae77a1-5934-42d2-8e88-d93e15356386-istio-podinfo\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.600144 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.600119    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcae77a1-5934-42d2-8e88-d93e15356386-istiod-ca-cert\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.600192 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.600149    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-istio-data\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.600250 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.600235    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-workload-certs\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.600289 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.600275    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-workload-socket\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.600335 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.600295    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcae77a1-5934-42d2-8e88-d93e15356386-istio-token\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.600442 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.600417    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-istio-envoy\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.600507 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.600490    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-credential-socket\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.701404 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.701371    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-istio-envoy\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.701404 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.701407    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-credential-socket\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.701650 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.701436    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmjwj\" (UniqueName: \"kubernetes.io/projected/fcae77a1-5934-42d2-8e88-d93e15356386-kube-api-access-mmjwj\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.701650 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.701469    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcae77a1-5934-42d2-8e88-d93e15356386-istio-podinfo\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.701650 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.701486    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcae77a1-5934-42d2-8e88-d93e15356386-istiod-ca-cert\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.701650 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.701505    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-istio-data\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.701650 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.701525    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-workload-certs\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.701650 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.701554    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-workload-socket\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.701650 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.701625    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcae77a1-5934-42d2-8e88-d93e15356386-istio-token\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.702133 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.702081    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-credential-socket\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.702133 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.702100    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-workload-certs\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.702434 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.702410    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-workload-socket\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.703036 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.703011    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcae77a1-5934-42d2-8e88-d93e15356386-istiod-ca-cert\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.703291 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.703271    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-istio-data\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.704313 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.704289    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-istio-envoy\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.704817 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.704796    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcae77a1-5934-42d2-8e88-d93e15356386-istio-podinfo\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.758907 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.758840    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcae77a1-5934-42d2-8e88-d93e15356386-istio-token\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.759122 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.759012    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmjwj\" (UniqueName: \"kubernetes.io/projected/fcae77a1-5934-42d2-8e88-d93e15356386-kube-api-access-mmjwj\") pod \"e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") " pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:24.808102 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:24.808068    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:25.233572 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:25.233536    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"]
Oct 31 00:05:25.271033 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:05:25.271006    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcae77a1_5934_42d2_8e88_d93e15356386.slice/crio-a5d1894f17ddf90db7bcf1a5abc1b30af82b41da81c1f2c64f9fe3c09641a849 WatchSource:0}: Error finding container a5d1894f17ddf90db7bcf1a5abc1b30af82b41da81c1f2c64f9fe3c09641a849: Status 404 returned error can't find the container with id a5d1894f17ddf90db7bcf1a5abc1b30af82b41da81c1f2c64f9fe3c09641a849
Oct 31 00:05:25.273028 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:25.272995    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:05:25.273114 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:25.273064    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:05:25.273114 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:25.273093    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:05:26.183245 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:26.183209    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr" event={"ID":"fcae77a1-5934-42d2-8e88-d93e15356386","Type":"ContainerStarted","Data":"5df49bc40183f1c450ce6e9997228c0275a4e5d3ea78e9bb3217ff94c01d5c34"}
Oct 31 00:05:26.183751 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:26.183263    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr" event={"ID":"fcae77a1-5934-42d2-8e88-d93e15356386","Type":"ContainerStarted","Data":"a5d1894f17ddf90db7bcf1a5abc1b30af82b41da81c1f2c64f9fe3c09641a849"}
Oct 31 00:05:26.808209 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:26.808167    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:26.813056 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:26.813028    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:26.930154 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:26.930102    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr" podStartSLOduration=2.930086393 podStartE2EDuration="2.930086393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:05:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:05:26.398643763 +0000 UTC m=+3512.286550120" watchObservedRunningTime="2026-10-31 00:05:26.930086393 +0000 UTC m=+3512.817992745"
Oct 31 00:05:27.188093 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:27.188016    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:27.189025 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:27.189002    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:32.998681 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:32.998643    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6"]
Oct 31 00:05:33.002324 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:33.002306    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6"
Oct 31 00:05:33.054776 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:33.054736    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-9b1704\""
Oct 31 00:05:33.084343 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:33.084303    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfmth\" (UniqueName: \"kubernetes.io/projected/7c585e00-dab8-480a-9359-5a36166e5071-kube-api-access-wfmth\") pod \"payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6\" (UID: \"7c585e00-dab8-480a-9359-5a36166e5071\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6"
Oct 31 00:05:33.084521 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:33.084351    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c585e00-dab8-480a-9359-5a36166e5071-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6\" (UID: \"7c585e00-dab8-480a-9359-5a36166e5071\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6"
Oct 31 00:05:33.104428 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:33.104397    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-bd0c2d3e-dockercfg-tqpx7\""
Oct 31 00:05:33.106422 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:33.106399    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6"]
Oct 31 00:05:33.185825 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:33.185787    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmth\" (UniqueName: \"kubernetes.io/projected/7c585e00-dab8-480a-9359-5a36166e5071-kube-api-access-wfmth\") pod \"payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6\" (UID: \"7c585e00-dab8-480a-9359-5a36166e5071\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6"
Oct 31 00:05:33.186037 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:33.185831    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c585e00-dab8-480a-9359-5a36166e5071-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6\" (UID: \"7c585e00-dab8-480a-9359-5a36166e5071\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6"
Oct 31 00:05:33.186591 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:33.186564    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c585e00-dab8-480a-9359-5a36166e5071-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6\" (UID: \"7c585e00-dab8-480a-9359-5a36166e5071\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6"
Oct 31 00:05:33.243985 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:33.243922    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfmth\" (UniqueName: \"kubernetes.io/projected/7c585e00-dab8-480a-9359-5a36166e5071-kube-api-access-wfmth\") pod \"payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6\" (UID: \"7c585e00-dab8-480a-9359-5a36166e5071\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6"
Oct 31 00:05:33.315671 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:33.315637    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6"
Oct 31 00:05:33.689924 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:33.689900    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6"]
Oct 31 00:05:33.691855 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:05:33.691828    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c585e00_dab8_480a_9359_5a36166e5071.slice/crio-adaaa7df8373097834c6a14d9bb7ce931039642306ad0f4448c0f771fc784c17 WatchSource:0}: Error finding container adaaa7df8373097834c6a14d9bb7ce931039642306ad0f4448c0f771fc784c17: Status 404 returned error can't find the container with id adaaa7df8373097834c6a14d9bb7ce931039642306ad0f4448c0f771fc784c17
Oct 31 00:05:34.221416 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:34.221375    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6" event={"ID":"7c585e00-dab8-480a-9359-5a36166e5071","Type":"ContainerStarted","Data":"1431c3862a4bb455f6c2442ad1bd2ad2a31d7ed29fffb142d8ebf2db9ffd96ba"}
Oct 31 00:05:34.221896 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:34.221444    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6" event={"ID":"7c585e00-dab8-480a-9359-5a36166e5071","Type":"ContainerStarted","Data":"adaaa7df8373097834c6a14d9bb7ce931039642306ad0f4448c0f771fc784c17"}
Oct 31 00:05:34.339115 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:34.339055    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6" podStartSLOduration=2.339040266 podStartE2EDuration="2.339040266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:05:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:05:34.337839366 +0000 UTC m=+3520.225745729" watchObservedRunningTime="2026-10-31 00:05:34.339040266 +0000 UTC m=+3520.226946616"
Oct 31 00:05:35.232852 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:35.232809    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6"
Oct 31 00:05:38.052999 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:38.052962    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn"]
Oct 31 00:05:38.102218 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:38.102186    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-psdgn"]
Oct 31 00:05:38.296869 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:38.296835    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b6a3b8-bd2d-445d-b368-d7c7ef375882" path="/var/lib/kubelet/pods/75b6a3b8-bd2d-445d-b368-d7c7ef375882/volumes"
Oct 31 00:05:46.240268 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:46.240235    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6"
Oct 31 00:05:47.608491 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:05:47.608453    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:05:50.050675 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:50.050635    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x"]
Oct 31 00:05:50.100273 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:50.100230    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-v2k6x"]
Oct 31 00:05:50.297910 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:50.297865    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b" path="/var/lib/kubelet/pods/da9a6c60-f4f2-4b3b-90ad-66bba07c5b3b/volumes"
Oct 31 00:05:53.328738 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:53.328702    5078 generic.go:356] "Generic (PLEG): container finished" podID="fcae77a1-5934-42d2-8e88-d93e15356386" containerID="5df49bc40183f1c450ce6e9997228c0275a4e5d3ea78e9bb3217ff94c01d5c34" exitCode=0
Oct 31 00:05:53.329190 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:53.328782    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr" event={"ID":"fcae77a1-5934-42d2-8e88-d93e15356386","Type":"ContainerDied","Data":"5df49bc40183f1c450ce6e9997228c0275a4e5d3ea78e9bb3217ff94c01d5c34"}
Oct 31 00:05:53.329190 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:53.329174    5078 scope.go:122] "RemoveContainer" containerID="5df49bc40183f1c450ce6e9997228c0275a4e5d3ea78e9bb3217ff94c01d5c34"
Oct 31 00:05:53.330562 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:53.330521    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:05:53.330661 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:53.330638    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:05:53.330706 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:53.330670    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:05:54.335211 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:54.335182    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr" event={"ID":"fcae77a1-5934-42d2-8e88-d93e15356386","Type":"ContainerStarted","Data":"391032ac545b02a3bdbda682cc398214471cf8ca296c22aacb32fbc1e63b2235"}
Oct 31 00:05:54.809251 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:54.809214    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:54.810479 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:54.810457    5078 patch_prober.go:29] interesting pod/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 31 00:05:54.810553 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:54.810507    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr" podUID="fcae77a1-5934-42d2-8e88-d93e15356386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 31 00:05:55.809278 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:55.809237    5078 patch_prober.go:29] interesting pod/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 31 00:05:55.809763 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:55.809295    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr" podUID="fcae77a1-5934-42d2-8e88-d93e15356386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 31 00:05:56.809082 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:56.809044    5078 patch_prober.go:29] interesting pod/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 31 00:05:56.809310 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:56.809113    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr" podUID="fcae77a1-5934-42d2-8e88-d93e15356386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 31 00:05:57.188577 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:57.188486    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:57.809203 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:57.809167    5078 patch_prober.go:29] interesting pod/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 31 00:05:57.809462 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:57.809229    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr" podUID="fcae77a1-5934-42d2-8e88-d93e15356386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 31 00:05:58.808460 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:58.808426    5078 patch_prober.go:29] interesting pod/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused" start-of-body=
Oct 31 00:05:58.808703 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:58.808486    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr" podUID="fcae77a1-5934-42d2-8e88-d93e15356386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.21:15021/healthz/ready\": dial tcp 10.133.1.21:15021: connect: connection refused"
Oct 31 00:05:59.812828 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:59.812798    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:05:59.814173 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:05:59.814150    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:06:17.590410 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:06:17.590369    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:06:25.047894 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:25.047861    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx"]
Oct 31 00:06:25.097469 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:25.097434    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-4rhdx"]
Oct 31 00:06:26.296799 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:26.296764    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d969ae-4db2-4b2c-a00c-f8de742bf25d" path="/var/lib/kubelet/pods/61d969ae-4db2-4b2c-a00c-f8de742bf25d/volumes"
Oct 31 00:06:37.046288 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:37.046254    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9"]
Oct 31 00:06:37.095583 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:37.095553    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-kj6g9"]
Oct 31 00:06:38.296710 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:38.296675    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f25ff8ed-973b-4c27-a0be-19b9fbd30f9d" path="/var/lib/kubelet/pods/f25ff8ed-973b-4c27-a0be-19b9fbd30f9d/volumes"
Oct 31 00:06:41.744022 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:41.743990    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn"]
Oct 31 00:06:41.747505 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:41.747489    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn"
Oct 31 00:06:41.804920 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:41.804878    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn"]
Oct 31 00:06:41.851635 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:41.851577    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:06:41.851821 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:41.851682    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:06:41.901425 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:41.901383    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:06:41.901425 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:41.901389    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:06:41.934320 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:41.934291    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnxrl\" (UniqueName: \"kubernetes.io/projected/f5831912-d0b7-4878-9187-560c8f9627a4-kube-api-access-vnxrl\") pod \"maas-api-revoke-sub-01fa745c-gccqn\" (UID: \"f5831912-d0b7-4878-9187-560c8f9627a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn"
Oct 31 00:06:41.934510 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:41.934331    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5831912-d0b7-4878-9187-560c8f9627a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01fa745c-gccqn\" (UID: \"f5831912-d0b7-4878-9187-560c8f9627a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn"
Oct 31 00:06:41.934510 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:41.934357    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5831912-d0b7-4878-9187-560c8f9627a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01fa745c-gccqn\" (UID: \"f5831912-d0b7-4878-9187-560c8f9627a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn"
Oct 31 00:06:42.035073 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:42.034974    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5831912-d0b7-4878-9187-560c8f9627a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01fa745c-gccqn\" (UID: \"f5831912-d0b7-4878-9187-560c8f9627a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn"
Oct 31 00:06:42.035265 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:42.035093    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnxrl\" (UniqueName: \"kubernetes.io/projected/f5831912-d0b7-4878-9187-560c8f9627a4-kube-api-access-vnxrl\") pod \"maas-api-revoke-sub-01fa745c-gccqn\" (UID: \"f5831912-d0b7-4878-9187-560c8f9627a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn"
Oct 31 00:06:42.035265 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:42.035143    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5831912-d0b7-4878-9187-560c8f9627a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01fa745c-gccqn\" (UID: \"f5831912-d0b7-4878-9187-560c8f9627a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn"
Oct 31 00:06:42.035807 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:42.035778    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5831912-d0b7-4878-9187-560c8f9627a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01fa745c-gccqn\" (UID: \"f5831912-d0b7-4878-9187-560c8f9627a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn"
Oct 31 00:06:42.037667 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:42.037639    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5831912-d0b7-4878-9187-560c8f9627a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01fa745c-gccqn\" (UID: \"f5831912-d0b7-4878-9187-560c8f9627a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn"
Oct 31 00:06:42.092409 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:42.092370    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnxrl\" (UniqueName: \"kubernetes.io/projected/f5831912-d0b7-4878-9187-560c8f9627a4-kube-api-access-vnxrl\") pod \"maas-api-revoke-sub-01fa745c-gccqn\" (UID: \"f5831912-d0b7-4878-9187-560c8f9627a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn"
Oct 31 00:06:42.359642 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:42.359588    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn"
Oct 31 00:06:42.729865 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:42.729771    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn"]
Oct 31 00:06:42.732137 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:06:42.732106    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5831912_d0b7_4878_9187_560c8f9627a4.slice/crio-863eda2230d88f4ab29626408af06fb055c653ebc54fdf80c3a120a9579f3b42 WatchSource:0}: Error finding container 863eda2230d88f4ab29626408af06fb055c653ebc54fdf80c3a120a9579f3b42: Status 404 returned error can't find the container with id 863eda2230d88f4ab29626408af06fb055c653ebc54fdf80c3a120a9579f3b42
Oct 31 00:06:43.569570 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:43.569529    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn" event={"ID":"f5831912-d0b7-4878-9187-560c8f9627a4","Type":"ContainerStarted","Data":"2255a6fd0f642ed0a3603d26fb27c011ee7b2063185ba8905eee4f8ad2c017e2"}
Oct 31 00:06:43.570042 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:43.569583    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn" event={"ID":"f5831912-d0b7-4878-9187-560c8f9627a4","Type":"ContainerStarted","Data":"863eda2230d88f4ab29626408af06fb055c653ebc54fdf80c3a120a9579f3b42"}
Oct 31 00:06:43.686887 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:43.686839    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn" podStartSLOduration=2.6868226330000002 podStartE2EDuration="2.686822633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:43.684794671 +0000 UTC m=+3589.572701024" watchObservedRunningTime="2026-10-31 00:06:43.686822633 +0000 UTC m=+3589.574729015"
Oct 31 00:06:44.575581 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:44.575539    5078 generic.go:356] "Generic (PLEG): container finished" podID="f5831912-d0b7-4878-9187-560c8f9627a4" containerID="2255a6fd0f642ed0a3603d26fb27c011ee7b2063185ba8905eee4f8ad2c017e2" exitCode=0
Oct 31 00:06:44.576082 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:44.575640    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn" event={"ID":"f5831912-d0b7-4878-9187-560c8f9627a4","Type":"ContainerDied","Data":"2255a6fd0f642ed0a3603d26fb27c011ee7b2063185ba8905eee4f8ad2c017e2"}
Oct 31 00:06:45.714500 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:45.714468    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn"
Oct 31 00:06:45.767805 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:45.767770    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5831912-d0b7-4878-9187-560c8f9627a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5831912-d0b7-4878-9187-560c8f9627a4-maas-api-xks-ca\") pod \"f5831912-d0b7-4878-9187-560c8f9627a4\" (UID: \"f5831912-d0b7-4878-9187-560c8f9627a4\") "
Oct 31 00:06:45.767972 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:45.767846    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5831912-d0b7-4878-9187-560c8f9627a4-kube-api-access-vnxrl\" (UniqueName: \"kubernetes.io/projected/f5831912-d0b7-4878-9187-560c8f9627a4-kube-api-access-vnxrl\") pod \"f5831912-d0b7-4878-9187-560c8f9627a4\" (UID: \"f5831912-d0b7-4878-9187-560c8f9627a4\") "
Oct 31 00:06:45.767972 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:45.767882    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5831912-d0b7-4878-9187-560c8f9627a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5831912-d0b7-4878-9187-560c8f9627a4-maas-api-openshift-ca\") pod \"f5831912-d0b7-4878-9187-560c8f9627a4\" (UID: \"f5831912-d0b7-4878-9187-560c8f9627a4\") "
Oct 31 00:06:45.768254 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:45.768232    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5831912-d0b7-4878-9187-560c8f9627a4-maas-api-openshift-ca" pod "f5831912-d0b7-4878-9187-560c8f9627a4" (UID: "f5831912-d0b7-4878-9187-560c8f9627a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:06:45.769936 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:45.769912    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5831912-d0b7-4878-9187-560c8f9627a4-maas-api-xks-ca" pod "f5831912-d0b7-4878-9187-560c8f9627a4" (UID: "f5831912-d0b7-4878-9187-560c8f9627a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:06:45.770065 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:45.770046    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5831912-d0b7-4878-9187-560c8f9627a4-kube-api-access-vnxrl" pod "f5831912-d0b7-4878-9187-560c8f9627a4" (UID: "f5831912-d0b7-4878-9187-560c8f9627a4"). InnerVolumeSpecName "kube-api-access-vnxrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:06:45.868729 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:45.868648    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5831912-d0b7-4878-9187-560c8f9627a4-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:06:45.868729 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:45.868687    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnxrl\" (UniqueName: \"kubernetes.io/projected/f5831912-d0b7-4878-9187-560c8f9627a4-kube-api-access-vnxrl\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:06:45.868729 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:45.868699    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5831912-d0b7-4878-9187-560c8f9627a4-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:06:46.588437 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:46.588356    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn"
Oct 31 00:06:46.588437 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:46.588375    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn" event={"ID":"f5831912-d0b7-4878-9187-560c8f9627a4","Type":"ContainerDied","Data":"863eda2230d88f4ab29626408af06fb055c653ebc54fdf80c3a120a9579f3b42"}
Oct 31 00:06:46.588437 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:46.588420    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="863eda2230d88f4ab29626408af06fb055c653ebc54fdf80c3a120a9579f3b42"
Oct 31 00:06:47.590502 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:06:47.590462    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:06:53.990987 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:53.990951    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9"]
Oct 31 00:06:53.991502 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:53.991290    5078 cpu_manager.go:414] "containerMap: removing container" podUID="f5831912-d0b7-4878-9187-560c8f9627a4" containerName="revoke-keys"
Oct 31 00:06:53.991502 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:53.991301    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5831912-d0b7-4878-9187-560c8f9627a4" containerName="revoke-keys"
Oct 31 00:06:53.991502 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:53.991360    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5831912-d0b7-4878-9187-560c8f9627a4" containerName="revoke-keys"
Oct 31 00:06:53.994447 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:53.994431    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9"
Oct 31 00:06:54.038978 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:54.038949    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gctgn\" (UniqueName: \"kubernetes.io/projected/cf7a2640-378b-4e00-bf1a-50705228ea87-kube-api-access-gctgn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9\" (UID: \"cf7a2640-378b-4e00-bf1a-50705228ea87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9"
Oct 31 00:06:54.039126 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:54.038986    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7a2640-378b-4e00-bf1a-50705228ea87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9\" (UID: \"cf7a2640-378b-4e00-bf1a-50705228ea87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9"
Oct 31 00:06:54.039126 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:54.039075    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7a2640-378b-4e00-bf1a-50705228ea87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9\" (UID: \"cf7a2640-378b-4e00-bf1a-50705228ea87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9"
Oct 31 00:06:54.048953 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:54.048916    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:06:54.049540 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:54.049317    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:06:54.098867 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:54.098836    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:06:54.098867 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:54.098861    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:06:54.100733 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:54.100713    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9"]
Oct 31 00:06:54.139746 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:54.139719    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7a2640-378b-4e00-bf1a-50705228ea87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9\" (UID: \"cf7a2640-378b-4e00-bf1a-50705228ea87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9"
Oct 31 00:06:54.139881 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:54.139818    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gctgn\" (UniqueName: \"kubernetes.io/projected/cf7a2640-378b-4e00-bf1a-50705228ea87-kube-api-access-gctgn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9\" (UID: \"cf7a2640-378b-4e00-bf1a-50705228ea87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9"
Oct 31 00:06:54.139881 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:54.139845    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7a2640-378b-4e00-bf1a-50705228ea87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9\" (UID: \"cf7a2640-378b-4e00-bf1a-50705228ea87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9"
Oct 31 00:06:54.140341 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:54.140323    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7a2640-378b-4e00-bf1a-50705228ea87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9\" (UID: \"cf7a2640-378b-4e00-bf1a-50705228ea87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9"
Oct 31 00:06:54.142319 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:54.142300    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7a2640-378b-4e00-bf1a-50705228ea87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9\" (UID: \"cf7a2640-378b-4e00-bf1a-50705228ea87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9"
Oct 31 00:06:54.196588 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:54.196553    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gctgn\" (UniqueName: \"kubernetes.io/projected/cf7a2640-378b-4e00-bf1a-50705228ea87-kube-api-access-gctgn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9\" (UID: \"cf7a2640-378b-4e00-bf1a-50705228ea87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9"
Oct 31 00:06:54.358224 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:54.358196    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:06:54.367275 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:54.367247    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9"
Oct 31 00:06:54.739943 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:54.739916    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9"]
Oct 31 00:06:54.741826 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:06:54.741794    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7a2640_378b_4e00_bf1a_50705228ea87.slice/crio-5879d21034eba620fdab34b6742a43993360dab47c865b65ea3d06eb640f48aa WatchSource:0}: Error finding container 5879d21034eba620fdab34b6742a43993360dab47c865b65ea3d06eb640f48aa: Status 404 returned error can't find the container with id 5879d21034eba620fdab34b6742a43993360dab47c865b65ea3d06eb640f48aa
Oct 31 00:06:55.630763 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:55.630737    5078 generic.go:356] "Generic (PLEG): container finished" podID="cf7a2640-378b-4e00-bf1a-50705228ea87" containerID="70e9d987f60a0624799b713756687ed35b1be562c6acd2af585493972b0d0668" exitCode=0
Oct 31 00:06:55.631127 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:55.630812    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9" event={"ID":"cf7a2640-378b-4e00-bf1a-50705228ea87","Type":"ContainerDied","Data":"70e9d987f60a0624799b713756687ed35b1be562c6acd2af585493972b0d0668"}
Oct 31 00:06:55.631127 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:55.630877    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9" event={"ID":"cf7a2640-378b-4e00-bf1a-50705228ea87","Type":"ContainerStarted","Data":"5879d21034eba620fdab34b6742a43993360dab47c865b65ea3d06eb640f48aa"}
Oct 31 00:06:56.798796 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:56.798772    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9"
Oct 31 00:06:56.863708 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:56.863676    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf7a2640-378b-4e00-bf1a-50705228ea87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7a2640-378b-4e00-bf1a-50705228ea87-maas-api-xks-ca\") pod \"cf7a2640-378b-4e00-bf1a-50705228ea87\" (UID: \"cf7a2640-378b-4e00-bf1a-50705228ea87\") "
Oct 31 00:06:56.863904 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:56.863799    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf7a2640-378b-4e00-bf1a-50705228ea87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7a2640-378b-4e00-bf1a-50705228ea87-maas-api-openshift-ca\") pod \"cf7a2640-378b-4e00-bf1a-50705228ea87\" (UID: \"cf7a2640-378b-4e00-bf1a-50705228ea87\") "
Oct 31 00:06:56.863904 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:56.863820    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf7a2640-378b-4e00-bf1a-50705228ea87-kube-api-access-gctgn\" (UniqueName: \"kubernetes.io/projected/cf7a2640-378b-4e00-bf1a-50705228ea87-kube-api-access-gctgn\") pod \"cf7a2640-378b-4e00-bf1a-50705228ea87\" (UID: \"cf7a2640-378b-4e00-bf1a-50705228ea87\") "
Oct 31 00:06:56.864215 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:56.864180    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7a2640-378b-4e00-bf1a-50705228ea87-maas-api-openshift-ca" pod "cf7a2640-378b-4e00-bf1a-50705228ea87" (UID: "cf7a2640-378b-4e00-bf1a-50705228ea87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:06:56.865907 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:56.865884    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf7a2640-378b-4e00-bf1a-50705228ea87-maas-api-xks-ca" pod "cf7a2640-378b-4e00-bf1a-50705228ea87" (UID: "cf7a2640-378b-4e00-bf1a-50705228ea87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:06:56.866097 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:56.866073    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7a2640-378b-4e00-bf1a-50705228ea87-kube-api-access-gctgn" pod "cf7a2640-378b-4e00-bf1a-50705228ea87" (UID: "cf7a2640-378b-4e00-bf1a-50705228ea87"). InnerVolumeSpecName "kube-api-access-gctgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:06:56.965441 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:56.965365    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gctgn\" (UniqueName: \"kubernetes.io/projected/cf7a2640-378b-4e00-bf1a-50705228ea87-kube-api-access-gctgn\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:06:56.965441 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:56.965393    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf7a2640-378b-4e00-bf1a-50705228ea87-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:06:56.965441 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:56.965403    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf7a2640-378b-4e00-bf1a-50705228ea87-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:06:57.641191 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:57.641153    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9"
Oct 31 00:06:57.641375 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:57.641159    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9" event={"ID":"cf7a2640-378b-4e00-bf1a-50705228ea87","Type":"ContainerDied","Data":"5879d21034eba620fdab34b6742a43993360dab47c865b65ea3d06eb640f48aa"}
Oct 31 00:06:57.641375 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:06:57.641277    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5879d21034eba620fdab34b6742a43993360dab47c865b65ea3d06eb640f48aa"
Oct 31 00:07:04.257099 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:04.257023    5078 scope.go:122] "RemoveContainer" containerID="5cb13ea0463b913ebbff2573b19a3a8872e04bcafafed80d9cddb759cc0b6d22"
Oct 31 00:07:04.265625 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:04.265587    5078 scope.go:122] "RemoveContainer" containerID="739b588ea746c57ad7292a28dd7e62906f4c2a0ebbd2b14208691b01c2ff92b9"
Oct 31 00:07:05.499346 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:05.499311    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6"]
Oct 31 00:07:10.500526 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:10.500490    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6" podUID="7c585e00-dab8-480a-9359-5a36166e5071" containerName="payload-processing" containerID="cri-o://1431c3862a4bb455f6c2442ad1bd2ad2a31d7ed29fffb142d8ebf2db9ffd96ba" gracePeriod=25
Oct 31 00:07:14.045407 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:14.045374    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn"]
Oct 31 00:07:14.095297 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:14.095270    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-2rcqn"]
Oct 31 00:07:14.296001 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:14.295974    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea35cc93-a1e8-4a3f-93b5-2708a5027762" path="/var/lib/kubelet/pods/ea35cc93-a1e8-4a3f-93b5-2708a5027762/volumes"
Oct 31 00:07:14.973233 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:14.973192    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9"]
Oct 31 00:07:15.022787 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:15.022760    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-9b1704-prhh9"]
Oct 31 00:07:16.239832 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:16.239797    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6" podUID="7c585e00-dab8-480a-9359-5a36166e5071" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:07:16.296509 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:16.296474    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf7a2640-378b-4e00-bf1a-50705228ea87" path="/var/lib/kubelet/pods/cf7a2640-378b-4e00-bf1a-50705228ea87/volumes"
Oct 31 00:07:17.605255 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:07:17.605186    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:07:21.419357 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:21.419326    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"]
Oct 31 00:07:21.419844 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:21.419556    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr" podUID="fcae77a1-5934-42d2-8e88-d93e15356386" containerName="istio-proxy" containerID="cri-o://391032ac545b02a3bdbda682cc398214471cf8ca296c22aacb32fbc1e63b2235" gracePeriod=30
Oct 31 00:07:22.941209 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:22.941173    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2"]
Oct 31 00:07:22.990588 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:22.990551    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-7kcg2"]
Oct 31 00:07:23.065412 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.065378    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7"]
Oct 31 00:07:23.065789 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.065774    5078 cpu_manager.go:414] "containerMap: removing container" podUID="cf7a2640-378b-4e00-bf1a-50705228ea87" containerName="revoke-keys"
Oct 31 00:07:23.065789 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.065789    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf7a2640-378b-4e00-bf1a-50705228ea87" containerName="revoke-keys"
Oct 31 00:07:23.065955 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.065854    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf7a2640-378b-4e00-bf1a-50705228ea87" containerName="revoke-keys"
Oct 31 00:07:23.070444 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.070422    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7"
Oct 31 00:07:23.123185 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.123156    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:07:23.123427 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.123178    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:07:23.123427 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.123229    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:07:23.123427 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.123228    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:07:23.126001 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.125971    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7"]
Oct 31 00:07:23.223986 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.223892    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmj2k\" (UniqueName: \"kubernetes.io/projected/371a9eb4-51eb-4d3d-b1f7-462d0321828b-kube-api-access-vmj2k\") pod \"maas-api-revoke-sub-3d4156f1-9mnq7\" (UID: \"371a9eb4-51eb-4d3d-b1f7-462d0321828b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7"
Oct 31 00:07:23.223986 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.223940    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371a9eb4-51eb-4d3d-b1f7-462d0321828b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9mnq7\" (UID: \"371a9eb4-51eb-4d3d-b1f7-462d0321828b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7"
Oct 31 00:07:23.224174 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.224069    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371a9eb4-51eb-4d3d-b1f7-462d0321828b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9mnq7\" (UID: \"371a9eb4-51eb-4d3d-b1f7-462d0321828b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7"
Oct 31 00:07:23.325430 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.325389    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371a9eb4-51eb-4d3d-b1f7-462d0321828b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9mnq7\" (UID: \"371a9eb4-51eb-4d3d-b1f7-462d0321828b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7"
Oct 31 00:07:23.325669 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.325471    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmj2k\" (UniqueName: \"kubernetes.io/projected/371a9eb4-51eb-4d3d-b1f7-462d0321828b-kube-api-access-vmj2k\") pod \"maas-api-revoke-sub-3d4156f1-9mnq7\" (UID: \"371a9eb4-51eb-4d3d-b1f7-462d0321828b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7"
Oct 31 00:07:23.325669 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.325507    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371a9eb4-51eb-4d3d-b1f7-462d0321828b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9mnq7\" (UID: \"371a9eb4-51eb-4d3d-b1f7-462d0321828b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7"
Oct 31 00:07:23.326200 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.326177    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371a9eb4-51eb-4d3d-b1f7-462d0321828b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9mnq7\" (UID: \"371a9eb4-51eb-4d3d-b1f7-462d0321828b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7"
Oct 31 00:07:23.328017 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.327993    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371a9eb4-51eb-4d3d-b1f7-462d0321828b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9mnq7\" (UID: \"371a9eb4-51eb-4d3d-b1f7-462d0321828b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7"
Oct 31 00:07:23.382446 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.382410    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmj2k\" (UniqueName: \"kubernetes.io/projected/371a9eb4-51eb-4d3d-b1f7-462d0321828b-kube-api-access-vmj2k\") pod \"maas-api-revoke-sub-3d4156f1-9mnq7\" (UID: \"371a9eb4-51eb-4d3d-b1f7-462d0321828b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7"
Oct 31 00:07:23.383039 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.383021    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7"
Oct 31 00:07:23.757328 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:23.757305    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7"]
Oct 31 00:07:23.760019 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:07:23.759988    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371a9eb4_51eb_4d3d_b1f7_462d0321828b.slice/crio-c80c64da2b8fa9d16cb2da921f10f3a48be4f6b35333d4c42ce3db042e0cbdd2 WatchSource:0}: Error finding container c80c64da2b8fa9d16cb2da921f10f3a48be4f6b35333d4c42ce3db042e0cbdd2: Status 404 returned error can't find the container with id c80c64da2b8fa9d16cb2da921f10f3a48be4f6b35333d4c42ce3db042e0cbdd2
Oct 31 00:07:24.296752 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:24.296719    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fad9310-270a-46ba-853f-02c682398111" path="/var/lib/kubelet/pods/1fad9310-270a-46ba-853f-02c682398111/volumes"
Oct 31 00:07:24.765474 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:24.765442    5078 generic.go:356] "Generic (PLEG): container finished" podID="371a9eb4-51eb-4d3d-b1f7-462d0321828b" containerID="cd317bb3aabdc757cd7700bde1ca04e573297113860cf5da9d28bdecb15508c1" exitCode=0
Oct 31 00:07:24.765694 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:24.765521    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7" event={"ID":"371a9eb4-51eb-4d3d-b1f7-462d0321828b","Type":"ContainerDied","Data":"cd317bb3aabdc757cd7700bde1ca04e573297113860cf5da9d28bdecb15508c1"}
Oct 31 00:07:24.765694 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:24.765570    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7" event={"ID":"371a9eb4-51eb-4d3d-b1f7-462d0321828b","Type":"ContainerStarted","Data":"c80c64da2b8fa9d16cb2da921f10f3a48be4f6b35333d4c42ce3db042e0cbdd2"}
Oct 31 00:07:25.905553 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:25.905525    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7"
Oct 31 00:07:26.049471 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.049432    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/371a9eb4-51eb-4d3d-b1f7-462d0321828b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371a9eb4-51eb-4d3d-b1f7-462d0321828b-maas-api-xks-ca\") pod \"371a9eb4-51eb-4d3d-b1f7-462d0321828b\" (UID: \"371a9eb4-51eb-4d3d-b1f7-462d0321828b\") "
Oct 31 00:07:26.049694 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.049583    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371a9eb4-51eb-4d3d-b1f7-462d0321828b-kube-api-access-vmj2k\" (UniqueName: \"kubernetes.io/projected/371a9eb4-51eb-4d3d-b1f7-462d0321828b-kube-api-access-vmj2k\") pod \"371a9eb4-51eb-4d3d-b1f7-462d0321828b\" (UID: \"371a9eb4-51eb-4d3d-b1f7-462d0321828b\") "
Oct 31 00:07:26.049694 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.049653    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371a9eb4-51eb-4d3d-b1f7-462d0321828b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371a9eb4-51eb-4d3d-b1f7-462d0321828b-maas-api-openshift-ca\") pod \"371a9eb4-51eb-4d3d-b1f7-462d0321828b\" (UID: \"371a9eb4-51eb-4d3d-b1f7-462d0321828b\") "
Oct 31 00:07:26.050055 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.050026    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371a9eb4-51eb-4d3d-b1f7-462d0321828b-maas-api-openshift-ca" pod "371a9eb4-51eb-4d3d-b1f7-462d0321828b" (UID: "371a9eb4-51eb-4d3d-b1f7-462d0321828b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:07:26.051905 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.051869    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371a9eb4-51eb-4d3d-b1f7-462d0321828b-kube-api-access-vmj2k" pod "371a9eb4-51eb-4d3d-b1f7-462d0321828b" (UID: "371a9eb4-51eb-4d3d-b1f7-462d0321828b"). InnerVolumeSpecName "kube-api-access-vmj2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:07:26.051905 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.051888    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/371a9eb4-51eb-4d3d-b1f7-462d0321828b-maas-api-xks-ca" pod "371a9eb4-51eb-4d3d-b1f7-462d0321828b" (UID: "371a9eb4-51eb-4d3d-b1f7-462d0321828b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:07:26.150568 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.150531    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmj2k\" (UniqueName: \"kubernetes.io/projected/371a9eb4-51eb-4d3d-b1f7-462d0321828b-kube-api-access-vmj2k\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:07:26.150568 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.150561    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/371a9eb4-51eb-4d3d-b1f7-462d0321828b-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:07:26.150568 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.150572    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/371a9eb4-51eb-4d3d-b1f7-462d0321828b-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:07:26.239123 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.239082    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6" podUID="7c585e00-dab8-480a-9359-5a36166e5071" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:07:26.672099 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.672074    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:07:26.777037 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.777005    5078 generic.go:356] "Generic (PLEG): container finished" podID="fcae77a1-5934-42d2-8e88-d93e15356386" containerID="391032ac545b02a3bdbda682cc398214471cf8ca296c22aacb32fbc1e63b2235" exitCode=0
Oct 31 00:07:26.777206 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.777059    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"
Oct 31 00:07:26.777206 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.777090    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr" event={"ID":"fcae77a1-5934-42d2-8e88-d93e15356386","Type":"ContainerDied","Data":"391032ac545b02a3bdbda682cc398214471cf8ca296c22aacb32fbc1e63b2235"}
Oct 31 00:07:26.777206 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.777161    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr" event={"ID":"fcae77a1-5934-42d2-8e88-d93e15356386","Type":"ContainerDied","Data":"a5d1894f17ddf90db7bcf1a5abc1b30af82b41da81c1f2c64f9fe3c09641a849"}
Oct 31 00:07:26.777206 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.777186    5078 scope.go:122] "RemoveContainer" containerID="391032ac545b02a3bdbda682cc398214471cf8ca296c22aacb32fbc1e63b2235"
Oct 31 00:07:26.778543 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.778340    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7" event={"ID":"371a9eb4-51eb-4d3d-b1f7-462d0321828b","Type":"ContainerDied","Data":"c80c64da2b8fa9d16cb2da921f10f3a48be4f6b35333d4c42ce3db042e0cbdd2"}
Oct 31 00:07:26.778543 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.778380    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7"
Oct 31 00:07:26.778543 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.778379    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c80c64da2b8fa9d16cb2da921f10f3a48be4f6b35333d4c42ce3db042e0cbdd2"
Oct 31 00:07:26.785986 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.785968    5078 scope.go:122] "RemoveContainer" containerID="5df49bc40183f1c450ce6e9997228c0275a4e5d3ea78e9bb3217ff94c01d5c34"
Oct 31 00:07:26.794285 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.794265    5078 scope.go:122] "RemoveContainer" containerID="391032ac545b02a3bdbda682cc398214471cf8ca296c22aacb32fbc1e63b2235"
Oct 31 00:07:26.794625 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:07:26.794576    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391032ac545b02a3bdbda682cc398214471cf8ca296c22aacb32fbc1e63b2235\": container with ID starting with 391032ac545b02a3bdbda682cc398214471cf8ca296c22aacb32fbc1e63b2235 not found: ID does not exist" containerID="391032ac545b02a3bdbda682cc398214471cf8ca296c22aacb32fbc1e63b2235"
Oct 31 00:07:26.794688 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.794625    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391032ac545b02a3bdbda682cc398214471cf8ca296c22aacb32fbc1e63b2235"} err="failed to get container status \"391032ac545b02a3bdbda682cc398214471cf8ca296c22aacb32fbc1e63b2235\": rpc error: code = NotFound desc = could not find container \"391032ac545b02a3bdbda682cc398214471cf8ca296c22aacb32fbc1e63b2235\": container with ID starting with 391032ac545b02a3bdbda682cc398214471cf8ca296c22aacb32fbc1e63b2235 not found: ID does not exist"
Oct 31 00:07:26.794688 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.794645    5078 scope.go:122] "RemoveContainer" containerID="5df49bc40183f1c450ce6e9997228c0275a4e5d3ea78e9bb3217ff94c01d5c34"
Oct 31 00:07:26.794913 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:07:26.794894    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df49bc40183f1c450ce6e9997228c0275a4e5d3ea78e9bb3217ff94c01d5c34\": container with ID starting with 5df49bc40183f1c450ce6e9997228c0275a4e5d3ea78e9bb3217ff94c01d5c34 not found: ID does not exist" containerID="5df49bc40183f1c450ce6e9997228c0275a4e5d3ea78e9bb3217ff94c01d5c34"
Oct 31 00:07:26.794961 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.794920    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df49bc40183f1c450ce6e9997228c0275a4e5d3ea78e9bb3217ff94c01d5c34"} err="failed to get container status \"5df49bc40183f1c450ce6e9997228c0275a4e5d3ea78e9bb3217ff94c01d5c34\": rpc error: code = NotFound desc = could not find container \"5df49bc40183f1c450ce6e9997228c0275a4e5d3ea78e9bb3217ff94c01d5c34\": container with ID starting with 5df49bc40183f1c450ce6e9997228c0275a4e5d3ea78e9bb3217ff94c01d5c34 not found: ID does not exist"
Oct 31 00:07:26.857387 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.857307    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-credential-socket\") pod \"fcae77a1-5934-42d2-8e88-d93e15356386\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") "
Oct 31 00:07:26.857387 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.857368    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-workload-socket\") pod \"fcae77a1-5934-42d2-8e88-d93e15356386\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") "
Oct 31 00:07:26.857663 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.857408    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-istio-envoy\") pod \"fcae77a1-5934-42d2-8e88-d93e15356386\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") "
Oct 31 00:07:26.857663 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.857460    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcae77a1-5934-42d2-8e88-d93e15356386-istio-token\" (UniqueName: \"kubernetes.io/projected/fcae77a1-5934-42d2-8e88-d93e15356386-istio-token\") pod \"fcae77a1-5934-42d2-8e88-d93e15356386\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") "
Oct 31 00:07:26.857663 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.857500    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcae77a1-5934-42d2-8e88-d93e15356386-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcae77a1-5934-42d2-8e88-d93e15356386-istiod-ca-cert\") pod \"fcae77a1-5934-42d2-8e88-d93e15356386\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") "
Oct 31 00:07:26.857663 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.857533    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcae77a1-5934-42d2-8e88-d93e15356386-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcae77a1-5934-42d2-8e88-d93e15356386-istio-podinfo\") pod \"fcae77a1-5934-42d2-8e88-d93e15356386\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") "
Oct 31 00:07:26.857663 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.857591    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-istio-data\") pod \"fcae77a1-5934-42d2-8e88-d93e15356386\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") "
Oct 31 00:07:26.857923 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.857668    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-workload-certs\") pod \"fcae77a1-5934-42d2-8e88-d93e15356386\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") "
Oct 31 00:07:26.857923 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.857729    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcae77a1-5934-42d2-8e88-d93e15356386-kube-api-access-mmjwj\" (UniqueName: \"kubernetes.io/projected/fcae77a1-5934-42d2-8e88-d93e15356386-kube-api-access-mmjwj\") pod \"fcae77a1-5934-42d2-8e88-d93e15356386\" (UID: \"fcae77a1-5934-42d2-8e88-d93e15356386\") "
Oct 31 00:07:26.857923 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.857590    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-credential-socket" pod "fcae77a1-5934-42d2-8e88-d93e15356386" (UID: "fcae77a1-5934-42d2-8e88-d93e15356386"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:07:26.857923 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.857724    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-workload-socket" pod "fcae77a1-5934-42d2-8e88-d93e15356386" (UID: "fcae77a1-5934-42d2-8e88-d93e15356386"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:07:26.857923 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.857874    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-istio-data" pod "fcae77a1-5934-42d2-8e88-d93e15356386" (UID: "fcae77a1-5934-42d2-8e88-d93e15356386"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:07:26.858211 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.857986    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcae77a1-5934-42d2-8e88-d93e15356386-istiod-ca-cert" pod "fcae77a1-5934-42d2-8e88-d93e15356386" (UID: "fcae77a1-5934-42d2-8e88-d93e15356386"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:07:26.858211 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.858082    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcae77a1-5934-42d2-8e88-d93e15356386-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:07:26.858211 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.858101    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:07:26.858211 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.858116    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:07:26.858211 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.858130    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:07:26.858682 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.858501    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-workload-certs" pod "fcae77a1-5934-42d2-8e88-d93e15356386" (UID: "fcae77a1-5934-42d2-8e88-d93e15356386"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:07:26.860113 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.860083    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcae77a1-5934-42d2-8e88-d93e15356386-istio-podinfo" pod "fcae77a1-5934-42d2-8e88-d93e15356386" (UID: "fcae77a1-5934-42d2-8e88-d93e15356386"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:07:26.860253 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.860125    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-istio-envoy" pod "fcae77a1-5934-42d2-8e88-d93e15356386" (UID: "fcae77a1-5934-42d2-8e88-d93e15356386"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:07:26.860530 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.860514    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcae77a1-5934-42d2-8e88-d93e15356386-kube-api-access-mmjwj" pod "fcae77a1-5934-42d2-8e88-d93e15356386" (UID: "fcae77a1-5934-42d2-8e88-d93e15356386"). InnerVolumeSpecName "kube-api-access-mmjwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:07:26.860648 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.860567    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcae77a1-5934-42d2-8e88-d93e15356386-istio-token" pod "fcae77a1-5934-42d2-8e88-d93e15356386" (UID: "fcae77a1-5934-42d2-8e88-d93e15356386"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:07:26.959329 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.959296    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcae77a1-5934-42d2-8e88-d93e15356386-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:07:26.959329 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.959329    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:07:26.959803 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.959347    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmjwj\" (UniqueName: \"kubernetes.io/projected/fcae77a1-5934-42d2-8e88-d93e15356386-kube-api-access-mmjwj\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:07:26.959803 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.959362    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcae77a1-5934-42d2-8e88-d93e15356386-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:07:26.959803 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:26.959376    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcae77a1-5934-42d2-8e88-d93e15356386-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:07:27.196192 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:27.196103    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"]
Oct 31 00:07:27.199304 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:27.199277    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-9b1704-openshift-default-84b6f64889j9qsr"]
Oct 31 00:07:28.296445 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:28.296408    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcae77a1-5934-42d2-8e88-d93e15356386" path="/var/lib/kubelet/pods/fcae77a1-5934-42d2-8e88-d93e15356386/volumes"
Oct 31 00:07:30.743375 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:30.743349    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6"
Oct 31 00:07:30.791487 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:30.791462    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c585e00-dab8-480a-9359-5a36166e5071-kube-api-access-wfmth\" (UniqueName: \"kubernetes.io/projected/7c585e00-dab8-480a-9359-5a36166e5071-kube-api-access-wfmth\") pod \"7c585e00-dab8-480a-9359-5a36166e5071\" (UID: \"7c585e00-dab8-480a-9359-5a36166e5071\") "
Oct 31 00:07:30.791666 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:30.791503    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c585e00-dab8-480a-9359-5a36166e5071-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c585e00-dab8-480a-9359-5a36166e5071-plugins-config-volume\") pod \"7c585e00-dab8-480a-9359-5a36166e5071\" (UID: \"7c585e00-dab8-480a-9359-5a36166e5071\") "
Oct 31 00:07:30.791956 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:30.791929    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c585e00-dab8-480a-9359-5a36166e5071-plugins-config-volume" pod "7c585e00-dab8-480a-9359-5a36166e5071" (UID: "7c585e00-dab8-480a-9359-5a36166e5071"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:07:30.793787 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:30.793767    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c585e00-dab8-480a-9359-5a36166e5071-kube-api-access-wfmth" pod "7c585e00-dab8-480a-9359-5a36166e5071" (UID: "7c585e00-dab8-480a-9359-5a36166e5071"). InnerVolumeSpecName "kube-api-access-wfmth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:07:30.801547 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:30.801520    5078 generic.go:356] "Generic (PLEG): container finished" podID="7c585e00-dab8-480a-9359-5a36166e5071" containerID="1431c3862a4bb455f6c2442ad1bd2ad2a31d7ed29fffb142d8ebf2db9ffd96ba" exitCode=0
Oct 31 00:07:30.801681 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:30.801574    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6" event={"ID":"7c585e00-dab8-480a-9359-5a36166e5071","Type":"ContainerDied","Data":"1431c3862a4bb455f6c2442ad1bd2ad2a31d7ed29fffb142d8ebf2db9ffd96ba"}
Oct 31 00:07:30.801681 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:30.801577    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6"
Oct 31 00:07:30.801681 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:30.801630    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6" event={"ID":"7c585e00-dab8-480a-9359-5a36166e5071","Type":"ContainerDied","Data":"adaaa7df8373097834c6a14d9bb7ce931039642306ad0f4448c0f771fc784c17"}
Oct 31 00:07:30.801681 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:30.801651    5078 scope.go:122] "RemoveContainer" containerID="1431c3862a4bb455f6c2442ad1bd2ad2a31d7ed29fffb142d8ebf2db9ffd96ba"
Oct 31 00:07:30.810141 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:30.810120    5078 scope.go:122] "RemoveContainer" containerID="1431c3862a4bb455f6c2442ad1bd2ad2a31d7ed29fffb142d8ebf2db9ffd96ba"
Oct 31 00:07:30.810426 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:07:30.810403    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1431c3862a4bb455f6c2442ad1bd2ad2a31d7ed29fffb142d8ebf2db9ffd96ba\": container with ID starting with 1431c3862a4bb455f6c2442ad1bd2ad2a31d7ed29fffb142d8ebf2db9ffd96ba not found: ID does not exist" containerID="1431c3862a4bb455f6c2442ad1bd2ad2a31d7ed29fffb142d8ebf2db9ffd96ba"
Oct 31 00:07:30.810472 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:30.810435    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1431c3862a4bb455f6c2442ad1bd2ad2a31d7ed29fffb142d8ebf2db9ffd96ba"} err="failed to get container status \"1431c3862a4bb455f6c2442ad1bd2ad2a31d7ed29fffb142d8ebf2db9ffd96ba\": rpc error: code = NotFound desc = could not find container \"1431c3862a4bb455f6c2442ad1bd2ad2a31d7ed29fffb142d8ebf2db9ffd96ba\": container with ID starting with 1431c3862a4bb455f6c2442ad1bd2ad2a31d7ed29fffb142d8ebf2db9ffd96ba not found: ID does not exist"
Oct 31 00:07:30.892396 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:30.892362    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfmth\" (UniqueName: \"kubernetes.io/projected/7c585e00-dab8-480a-9359-5a36166e5071-kube-api-access-wfmth\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:07:30.892396 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:30.892391    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c585e00-dab8-480a-9359-5a36166e5071-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:07:30.966806 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:30.966772    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6"]
Oct 31 00:07:30.972047 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:30.972023    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-9b1704-ccdbf496f4s5n6"]
Oct 31 00:07:32.296871 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:32.296838    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c585e00-dab8-480a-9359-5a36166e5071" path="/var/lib/kubelet/pods/7c585e00-dab8-480a-9359-5a36166e5071/volumes"
Oct 31 00:07:43.047092 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:43.047057    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2"]
Oct 31 00:07:43.096359 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:43.096330    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-blzg2"]
Oct 31 00:07:44.296652 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:07:44.296615    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10528c4-c8fd-41a1-b028-a82e08997577" path="/var/lib/kubelet/pods/f10528c4-c8fd-41a1-b028-a82e08997577/volumes"
Oct 31 00:07:47.590450 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:07:47.590417    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:08:04.287922 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:08:04.287890    5078 scope.go:122] "RemoveContainer" containerID="6d2e110c60b0fff7ef0548162007c8b5638348a97625400ac2f73aa35fd7fdcf"
Oct 31 00:08:04.299464 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:08:04.299105    5078 scope.go:122] "RemoveContainer" containerID="558674bb1f65fa341136b4bc9195bf2e79fe8005dc65082ba24966a23c3d0d89"
Oct 31 00:08:13.046330 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:08:13.046284    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744"]
Oct 31 00:08:13.095853 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:08:13.095819    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-gx744"]
Oct 31 00:08:14.301777 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:08:14.301742    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9" path="/var/lib/kubelet/pods/dcfca9d2-aae8-4f55-9ec2-44e53e9ccae9/volumes"
Oct 31 00:08:17.590680 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:08:17.590645    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:08:37.043666 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:08:37.043542    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l"]
Oct 31 00:08:37.093255 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:08:37.093219    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-frd7l"]
Oct 31 00:08:38.296491 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:08:38.296458    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ac1a712-bb7e-4680-8f22-20c1b19d38f5" path="/var/lib/kubelet/pods/1ac1a712-bb7e-4680-8f22-20c1b19d38f5/volumes"
Oct 31 00:08:47.590898 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:08:47.590855    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:09:04.328341 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:04.328307    5078 scope.go:122] "RemoveContainer" containerID="ae4674ce103504dcb1ec48aa783a1a0a5bc0303dbcd9321eeee0e14b4500c0a3"
Oct 31 00:09:04.336591 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:04.336559    5078 scope.go:122] "RemoveContainer" containerID="6a8c582db785030bde85b4653f04ee1ad6a7f96f21620590bcbb8da0792539f2"
Oct 31 00:09:12.984608 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:12.984569    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z"]
Oct 31 00:09:12.985129 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:12.984898    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7c585e00-dab8-480a-9359-5a36166e5071" containerName="payload-processing"
Oct 31 00:09:12.985129 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:12.984909    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c585e00-dab8-480a-9359-5a36166e5071" containerName="payload-processing"
Oct 31 00:09:12.985129 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:12.984924    5078 cpu_manager.go:414] "containerMap: removing container" podUID="fcae77a1-5934-42d2-8e88-d93e15356386" containerName="istio-proxy"
Oct 31 00:09:12.985129 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:12.984929    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcae77a1-5934-42d2-8e88-d93e15356386" containerName="istio-proxy"
Oct 31 00:09:12.985129 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:12.984944    5078 cpu_manager.go:414] "containerMap: removing container" podUID="fcae77a1-5934-42d2-8e88-d93e15356386" containerName="istio-proxy"
Oct 31 00:09:12.985129 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:12.984949    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcae77a1-5934-42d2-8e88-d93e15356386" containerName="istio-proxy"
Oct 31 00:09:12.985129 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:12.984961    5078 cpu_manager.go:414] "containerMap: removing container" podUID="371a9eb4-51eb-4d3d-b1f7-462d0321828b" containerName="revoke-keys"
Oct 31 00:09:12.985129 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:12.984965    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371a9eb4-51eb-4d3d-b1f7-462d0321828b" containerName="revoke-keys"
Oct 31 00:09:12.985129 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:12.985037    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="371a9eb4-51eb-4d3d-b1f7-462d0321828b" containerName="revoke-keys"
Oct 31 00:09:12.985129 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:12.985047    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c585e00-dab8-480a-9359-5a36166e5071" containerName="payload-processing"
Oct 31 00:09:12.985129 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:12.985053    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcae77a1-5934-42d2-8e88-d93e15356386" containerName="istio-proxy"
Oct 31 00:09:12.989430 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:12.989410    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z"
Oct 31 00:09:13.042051 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:13.042026    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:09:13.091914 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:13.091873    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:09:13.091914 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:13.091880    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:09:13.092130 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:13.091946    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:09:13.093463 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:13.093444    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z"]
Oct 31 00:09:13.147307 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:13.147269    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-bdx9z\" (UID: \"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z"
Oct 31 00:09:13.147505 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:13.147365    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd857\" (UniqueName: \"kubernetes.io/projected/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-kube-api-access-kd857\") pod \"maas-api-revoke-sub-d576e68b-bdx9z\" (UID: \"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z"
Oct 31 00:09:13.147505 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:13.147423    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-bdx9z\" (UID: \"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z"
Oct 31 00:09:13.248722 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:13.248632    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd857\" (UniqueName: \"kubernetes.io/projected/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-kube-api-access-kd857\") pod \"maas-api-revoke-sub-d576e68b-bdx9z\" (UID: \"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z"
Oct 31 00:09:13.248722 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:13.248674    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-bdx9z\" (UID: \"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z"
Oct 31 00:09:13.248722 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:13.248719    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-bdx9z\" (UID: \"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z"
Oct 31 00:09:13.249354 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:13.249328    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-bdx9z\" (UID: \"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z"
Oct 31 00:09:13.251192 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:13.251175    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-bdx9z\" (UID: \"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z"
Oct 31 00:09:13.306505 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:13.306458    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd857\" (UniqueName: \"kubernetes.io/projected/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-kube-api-access-kd857\") pod \"maas-api-revoke-sub-d576e68b-bdx9z\" (UID: \"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z"
Oct 31 00:09:13.601439 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:13.601402    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z"
Oct 31 00:09:13.970785 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:13.970756    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z"]
Oct 31 00:09:13.973099 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:09:13.973071    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1ee3ef_1403_4ee5_a61e_0bb9bfaabff2.slice/crio-80e72adeaa4d35ba927d8737148d1b5d2177c0b27384307a85eb35c7e19c8182 WatchSource:0}: Error finding container 80e72adeaa4d35ba927d8737148d1b5d2177c0b27384307a85eb35c7e19c8182: Status 404 returned error can't find the container with id 80e72adeaa4d35ba927d8737148d1b5d2177c0b27384307a85eb35c7e19c8182
Oct 31 00:09:14.305175 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:14.305143    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z" event={"ID":"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2","Type":"ContainerStarted","Data":"b0848c3431fab86b1bcbeee86eadd338d4147c56fb75f3e8782dc3d08c9da016"}
Oct 31 00:09:14.305585 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:14.305199    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z" event={"ID":"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2","Type":"ContainerStarted","Data":"80e72adeaa4d35ba927d8737148d1b5d2177c0b27384307a85eb35c7e19c8182"}
Oct 31 00:09:14.418984 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:14.418930    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z" podStartSLOduration=2.418913291 podStartE2EDuration="2.418913291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:14.417127589 +0000 UTC m=+3740.305033942" watchObservedRunningTime="2026-10-31 00:09:14.418913291 +0000 UTC m=+3740.306819644"
Oct 31 00:09:15.311081 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:15.311046    5078 generic.go:356] "Generic (PLEG): container finished" podID="0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2" containerID="b0848c3431fab86b1bcbeee86eadd338d4147c56fb75f3e8782dc3d08c9da016" exitCode=0
Oct 31 00:09:15.311534 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:15.311133    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z" event={"ID":"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2","Type":"ContainerDied","Data":"b0848c3431fab86b1bcbeee86eadd338d4147c56fb75f3e8782dc3d08c9da016"}
Oct 31 00:09:16.441331 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:16.441301    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z"
Oct 31 00:09:16.579929 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:16.579858    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-maas-api-openshift-ca\") pod \"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2\" (UID: \"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2\") "
Oct 31 00:09:16.580099 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:16.579960    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-maas-api-xks-ca\") pod \"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2\" (UID: \"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2\") "
Oct 31 00:09:16.580099 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:16.580055    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-kube-api-access-kd857\" (UniqueName: \"kubernetes.io/projected/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-kube-api-access-kd857\") pod \"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2\" (UID: \"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2\") "
Oct 31 00:09:16.580260 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:16.580240    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-maas-api-openshift-ca" pod "0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2" (UID: "0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:09:16.580360 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:16.580328    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:09:16.582831 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:16.582807    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-maas-api-xks-ca" pod "0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2" (UID: "0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:09:16.582831 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:16.582828    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-kube-api-access-kd857" pod "0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2" (UID: "0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2"). InnerVolumeSpecName "kube-api-access-kd857". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:09:16.680996 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:16.680969    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:09:16.680996 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:16.680994    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd857\" (UniqueName: \"kubernetes.io/projected/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2-kube-api-access-kd857\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:09:17.327968 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:17.327938    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z"
Oct 31 00:09:17.327968 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:17.327952    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z" event={"ID":"0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2","Type":"ContainerDied","Data":"80e72adeaa4d35ba927d8737148d1b5d2177c0b27384307a85eb35c7e19c8182"}
Oct 31 00:09:17.328193 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:17.328018    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80e72adeaa4d35ba927d8737148d1b5d2177c0b27384307a85eb35c7e19c8182"
Oct 31 00:09:17.590338 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:09:17.590255    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:09:41.966646 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:41.966587    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t"]
Oct 31 00:09:41.967157 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:41.966974    5078 cpu_manager.go:414] "containerMap: removing container" podUID="0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2" containerName="revoke-keys"
Oct 31 00:09:41.967157 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:41.966985    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2" containerName="revoke-keys"
Oct 31 00:09:41.967157 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:41.967037    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcae77a1-5934-42d2-8e88-d93e15356386" containerName="istio-proxy"
Oct 31 00:09:41.967157 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:41.967048    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2" containerName="revoke-keys"
Oct 31 00:09:41.970295 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:41.970270    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t"
Oct 31 00:09:42.026540 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:42.026495    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t"]
Oct 31 00:09:42.033258 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:42.033230    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xxln\" (UniqueName: \"kubernetes.io/projected/28092b6f-7204-4d92-82b7-3b4caece1cb9-kube-api-access-7xxln\") pod \"maas-api-revoke-sub-d2bc93bc-dbr6t\" (UID: \"28092b6f-7204-4d92-82b7-3b4caece1cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t"
Oct 31 00:09:42.033434 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:42.033290    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28092b6f-7204-4d92-82b7-3b4caece1cb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-dbr6t\" (UID: \"28092b6f-7204-4d92-82b7-3b4caece1cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t"
Oct 31 00:09:42.033434 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:42.033368    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28092b6f-7204-4d92-82b7-3b4caece1cb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-dbr6t\" (UID: \"28092b6f-7204-4d92-82b7-3b4caece1cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t"
Oct 31 00:09:42.074227 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:42.074193    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:09:42.074227 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:42.074205    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:09:42.074227 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:42.074207    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:09:42.074227 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:42.074230    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:09:42.133959 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:42.133923    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xxln\" (UniqueName: \"kubernetes.io/projected/28092b6f-7204-4d92-82b7-3b4caece1cb9-kube-api-access-7xxln\") pod \"maas-api-revoke-sub-d2bc93bc-dbr6t\" (UID: \"28092b6f-7204-4d92-82b7-3b4caece1cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t"
Oct 31 00:09:42.134350 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:42.134320    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28092b6f-7204-4d92-82b7-3b4caece1cb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-dbr6t\" (UID: \"28092b6f-7204-4d92-82b7-3b4caece1cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t"
Oct 31 00:09:42.134497 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:42.134377    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28092b6f-7204-4d92-82b7-3b4caece1cb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-dbr6t\" (UID: \"28092b6f-7204-4d92-82b7-3b4caece1cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t"
Oct 31 00:09:42.135041 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:42.135016    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28092b6f-7204-4d92-82b7-3b4caece1cb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-dbr6t\" (UID: \"28092b6f-7204-4d92-82b7-3b4caece1cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t"
Oct 31 00:09:42.137006 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:42.136982    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28092b6f-7204-4d92-82b7-3b4caece1cb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-dbr6t\" (UID: \"28092b6f-7204-4d92-82b7-3b4caece1cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t"
Oct 31 00:09:42.191325 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:42.191283    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xxln\" (UniqueName: \"kubernetes.io/projected/28092b6f-7204-4d92-82b7-3b4caece1cb9-kube-api-access-7xxln\") pod \"maas-api-revoke-sub-d2bc93bc-dbr6t\" (UID: \"28092b6f-7204-4d92-82b7-3b4caece1cb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t"
Oct 31 00:09:42.284720 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:42.284639    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t"
Oct 31 00:09:42.654453 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:42.654289    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t"]
Oct 31 00:09:42.656744 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:09:42.656715    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28092b6f_7204_4d92_82b7_3b4caece1cb9.slice/crio-43f7965a3da81dcad0e2ac4bc2fbd3f6b82c1abd483796f240f6c91d3dc5d0fb WatchSource:0}: Error finding container 43f7965a3da81dcad0e2ac4bc2fbd3f6b82c1abd483796f240f6c91d3dc5d0fb: Status 404 returned error can't find the container with id 43f7965a3da81dcad0e2ac4bc2fbd3f6b82c1abd483796f240f6c91d3dc5d0fb
Oct 31 00:09:43.472107 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:43.472070    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t" event={"ID":"28092b6f-7204-4d92-82b7-3b4caece1cb9","Type":"ContainerStarted","Data":"d8b7eb70953856104d2da949292c4e3c24a151034077235a144b0b564fd98629"}
Oct 31 00:09:43.472577 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:43.472136    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t" event={"ID":"28092b6f-7204-4d92-82b7-3b4caece1cb9","Type":"ContainerStarted","Data":"43f7965a3da81dcad0e2ac4bc2fbd3f6b82c1abd483796f240f6c91d3dc5d0fb"}
Oct 31 00:09:43.588859 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:43.588811    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t" podStartSLOduration=2.58879705 podStartE2EDuration="2.58879705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:43.588198387 +0000 UTC m=+3769.476104741" watchObservedRunningTime="2026-10-31 00:09:43.58879705 +0000 UTC m=+3769.476703394"
Oct 31 00:09:44.478357 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:44.478328    5078 generic.go:356] "Generic (PLEG): container finished" podID="28092b6f-7204-4d92-82b7-3b4caece1cb9" containerID="d8b7eb70953856104d2da949292c4e3c24a151034077235a144b0b564fd98629" exitCode=0
Oct 31 00:09:44.478892 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:44.478424    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t" event={"ID":"28092b6f-7204-4d92-82b7-3b4caece1cb9","Type":"ContainerDied","Data":"d8b7eb70953856104d2da949292c4e3c24a151034077235a144b0b564fd98629"}
Oct 31 00:09:45.045717 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:45.045680    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj"]
Oct 31 00:09:45.048099 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:45.048074    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qndcj"]
Oct 31 00:09:45.618613 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:45.618574    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t"
Oct 31 00:09:45.666519 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:45.666486    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28092b6f-7204-4d92-82b7-3b4caece1cb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28092b6f-7204-4d92-82b7-3b4caece1cb9-maas-api-xks-ca\") pod \"28092b6f-7204-4d92-82b7-3b4caece1cb9\" (UID: \"28092b6f-7204-4d92-82b7-3b4caece1cb9\") "
Oct 31 00:09:45.666679 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:45.666553    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28092b6f-7204-4d92-82b7-3b4caece1cb9-kube-api-access-7xxln\" (UniqueName: \"kubernetes.io/projected/28092b6f-7204-4d92-82b7-3b4caece1cb9-kube-api-access-7xxln\") pod \"28092b6f-7204-4d92-82b7-3b4caece1cb9\" (UID: \"28092b6f-7204-4d92-82b7-3b4caece1cb9\") "
Oct 31 00:09:45.666679 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:45.666622    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28092b6f-7204-4d92-82b7-3b4caece1cb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28092b6f-7204-4d92-82b7-3b4caece1cb9-maas-api-openshift-ca\") pod \"28092b6f-7204-4d92-82b7-3b4caece1cb9\" (UID: \"28092b6f-7204-4d92-82b7-3b4caece1cb9\") "
Oct 31 00:09:45.666976 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:45.666950    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28092b6f-7204-4d92-82b7-3b4caece1cb9-maas-api-openshift-ca" pod "28092b6f-7204-4d92-82b7-3b4caece1cb9" (UID: "28092b6f-7204-4d92-82b7-3b4caece1cb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:09:45.668891 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:45.668864    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28092b6f-7204-4d92-82b7-3b4caece1cb9-maas-api-xks-ca" pod "28092b6f-7204-4d92-82b7-3b4caece1cb9" (UID: "28092b6f-7204-4d92-82b7-3b4caece1cb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:09:45.669003 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:45.668869    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28092b6f-7204-4d92-82b7-3b4caece1cb9-kube-api-access-7xxln" pod "28092b6f-7204-4d92-82b7-3b4caece1cb9" (UID: "28092b6f-7204-4d92-82b7-3b4caece1cb9"). InnerVolumeSpecName "kube-api-access-7xxln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:09:45.767921 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:45.767841    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28092b6f-7204-4d92-82b7-3b4caece1cb9-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:09:45.767921 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:45.767875    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xxln\" (UniqueName: \"kubernetes.io/projected/28092b6f-7204-4d92-82b7-3b4caece1cb9-kube-api-access-7xxln\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:09:45.767921 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:45.767885    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28092b6f-7204-4d92-82b7-3b4caece1cb9-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:09:46.295979 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:46.295948    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9941253b-db94-4b2e-ba32-1d1034bb8a57" path="/var/lib/kubelet/pods/9941253b-db94-4b2e-ba32-1d1034bb8a57/volumes"
Oct 31 00:09:46.488690 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:46.488655    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t" event={"ID":"28092b6f-7204-4d92-82b7-3b4caece1cb9","Type":"ContainerDied","Data":"43f7965a3da81dcad0e2ac4bc2fbd3f6b82c1abd483796f240f6c91d3dc5d0fb"}
Oct 31 00:09:46.488904 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:46.488710    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43f7965a3da81dcad0e2ac4bc2fbd3f6b82c1abd483796f240f6c91d3dc5d0fb"
Oct 31 00:09:46.488904 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:46.488662    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t"
Oct 31 00:09:47.590583 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:09:47.590550    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:09:54.730115 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.730077    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"]
Oct 31 00:09:54.730655 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.730578    5078 cpu_manager.go:414] "containerMap: removing container" podUID="28092b6f-7204-4d92-82b7-3b4caece1cb9" containerName="revoke-keys"
Oct 31 00:09:54.730655 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.730620    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28092b6f-7204-4d92-82b7-3b4caece1cb9" containerName="revoke-keys"
Oct 31 00:09:54.730788 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.730724    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="28092b6f-7204-4d92-82b7-3b4caece1cb9" containerName="revoke-keys"
Oct 31 00:09:54.734059 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.734038    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.835916 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.835879    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"]
Oct 31 00:09:54.842376 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.842339    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-f5f39e-openshift-default-dockercfg-wf64j\""
Oct 31 00:09:54.852873 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.852839    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmhqs\" (UniqueName: \"kubernetes.io/projected/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-kube-api-access-gmhqs\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.853013 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.852903    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istiod-ca-cert\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.853071 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.853040    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-workload-socket\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.853133 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.853101    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-podinfo\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.853187 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.853152    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-envoy\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.853249 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.853195    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-data\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.853249 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.853231    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-workload-certs\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.853368 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.853286    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-token\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.853430 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.853412    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-credential-socket\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.954452 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.954412    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-workload-certs\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.954452 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.954464    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-token\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.954805 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.954490    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-credential-socket\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.954805 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.954534    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmhqs\" (UniqueName: \"kubernetes.io/projected/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-kube-api-access-gmhqs\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.954805 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.954738    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istiod-ca-cert\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.954805 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.954802    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-workload-socket\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.955077 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.954837    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-podinfo\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.955077 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.954889    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-workload-certs\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.955077 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.954896    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-envoy\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.955077 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.954940    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-credential-socket\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.955077 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.954963    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-data\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.955375 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.955143    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-workload-socket\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.955515 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.955491    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-data\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.955633 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.955514    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istiod-ca-cert\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.958014 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.957994    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-envoy\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:54.958161 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:54.958146    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-podinfo\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:55.011384 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:55.011309    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-token\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:55.011530 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:55.011435    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmhqs\" (UniqueName: \"kubernetes.io/projected/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-kube-api-access-gmhqs\") pod \"e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") " pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:55.045782 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:55.045750    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf"]
Oct 31 00:09:55.045947 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:55.045903    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:55.048576 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:55.048477    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9fvrf"]
Oct 31 00:09:55.506085 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:55.506062    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"]
Oct 31 00:09:55.508656 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:09:55.508628    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f84e47a_b5d9_4a4e_a2d3_fc3c5b46318d.slice/crio-ffb73197318a051f1ad7cdac8bbf965f69d5a8816c4cfb079662f4bb1caba59e WatchSource:0}: Error finding container ffb73197318a051f1ad7cdac8bbf965f69d5a8816c4cfb079662f4bb1caba59e: Status 404 returned error can't find the container with id ffb73197318a051f1ad7cdac8bbf965f69d5a8816c4cfb079662f4bb1caba59e
Oct 31 00:09:55.510737 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:55.510710    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:09:55.510825 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:55.510769    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:09:55.510825 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:55.510799    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:09:55.531786 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:55.531760    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs" event={"ID":"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d","Type":"ContainerStarted","Data":"ffb73197318a051f1ad7cdac8bbf965f69d5a8816c4cfb079662f4bb1caba59e"}
Oct 31 00:09:56.296698 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:56.296661    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ff93983-c7ab-4e70-a4c0-a43d66390539" path="/var/lib/kubelet/pods/5ff93983-c7ab-4e70-a4c0-a43d66390539/volumes"
Oct 31 00:09:56.538458 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:56.538414    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs" event={"ID":"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d","Type":"ContainerStarted","Data":"b8439ad535e287b53ce813e89ed248c8c1c521862195715b668418319db871c6"}
Oct 31 00:09:56.656015 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:56.655914    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs" podStartSLOduration=2.6558994609999997 podStartE2EDuration="2.655899461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:56.654361261 +0000 UTC m=+3782.542267616" watchObservedRunningTime="2026-10-31 00:09:56.655899461 +0000 UTC m=+3782.543805813"
Oct 31 00:09:57.046605 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:57.046567    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:57.051688 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:57.051662    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:57.543194 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:57.543173    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:09:57.544215 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:09:57.544198    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:10:03.234258 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:03.234173    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855"]
Oct 31 00:10:03.237569 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:03.237553    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855"
Oct 31 00:10:03.292740 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:03.292701    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-f5f39e\""
Oct 31 00:10:03.341026 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:03.340992    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855"]
Oct 31 00:10:03.361228 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:03.361190    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a0a9651-9b67-4834-a2e6-01d713044774-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855\" (UID: \"8a0a9651-9b67-4834-a2e6-01d713044774\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855"
Oct 31 00:10:03.361534 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:03.361503    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b75j4\" (UniqueName: \"kubernetes.io/projected/8a0a9651-9b67-4834-a2e6-01d713044774-kube-api-access-b75j4\") pod \"payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855\" (UID: \"8a0a9651-9b67-4834-a2e6-01d713044774\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855"
Oct 31 00:10:03.462783 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:03.462739    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a0a9651-9b67-4834-a2e6-01d713044774-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855\" (UID: \"8a0a9651-9b67-4834-a2e6-01d713044774\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855"
Oct 31 00:10:03.462982 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:03.462804    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b75j4\" (UniqueName: \"kubernetes.io/projected/8a0a9651-9b67-4834-a2e6-01d713044774-kube-api-access-b75j4\") pod \"payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855\" (UID: \"8a0a9651-9b67-4834-a2e6-01d713044774\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855"
Oct 31 00:10:03.463490 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:03.463464    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a0a9651-9b67-4834-a2e6-01d713044774-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855\" (UID: \"8a0a9651-9b67-4834-a2e6-01d713044774\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855"
Oct 31 00:10:03.520559 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:03.520490    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b75j4\" (UniqueName: \"kubernetes.io/projected/8a0a9651-9b67-4834-a2e6-01d713044774-kube-api-access-b75j4\") pod \"payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855\" (UID: \"8a0a9651-9b67-4834-a2e6-01d713044774\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855"
Oct 31 00:10:03.550464 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:03.550441    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855"
Oct 31 00:10:03.921309 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:03.921283    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855"]
Oct 31 00:10:03.923657 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:10:03.923582    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a0a9651_9b67_4834_a2e6_01d713044774.slice/crio-43108dee820a5a046ef0ce67b2311b3e48e866d8cbabc735eb6bc1ecf6e9f1fa WatchSource:0}: Error finding container 43108dee820a5a046ef0ce67b2311b3e48e866d8cbabc735eb6bc1ecf6e9f1fa: Status 404 returned error can't find the container with id 43108dee820a5a046ef0ce67b2311b3e48e866d8cbabc735eb6bc1ecf6e9f1fa
Oct 31 00:10:04.359076 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:04.359046    5078 scope.go:122] "RemoveContainer" containerID="11780bf3f29ea26f0e5441c7043c9839bba75cd3116e3257f2f5e306f36896fc"
Oct 31 00:10:04.367549 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:04.367519    5078 scope.go:122] "RemoveContainer" containerID="0ecd2b22ff0e147120bc10ecb15afd7de3116aa26be6ee576b1f8e4e619dc58d"
Oct 31 00:10:04.375553 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:04.375526    5078 scope.go:122] "RemoveContainer" containerID="9f810dc6d6f4d04b7e268ca852ff73b5df33b0dbb9ab90d55166f08579cf335e"
Oct 31 00:10:04.576913 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:04.576878    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855" event={"ID":"8a0a9651-9b67-4834-a2e6-01d713044774","Type":"ContainerStarted","Data":"983d6dddc4731e6f4e4c2cdf6153a92d217065fdb16e9c2a7ade461455a6b43b"}
Oct 31 00:10:04.577101 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:04.577033    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855"
Oct 31 00:10:04.577101 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:04.577071    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855" event={"ID":"8a0a9651-9b67-4834-a2e6-01d713044774","Type":"ContainerStarted","Data":"43108dee820a5a046ef0ce67b2311b3e48e866d8cbabc735eb6bc1ecf6e9f1fa"}
Oct 31 00:10:04.691026 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:04.690930    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855" podStartSLOduration=1.690915943 podStartE2EDuration="1.690915943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:04.689855861 +0000 UTC m=+3790.577762214" watchObservedRunningTime="2026-10-31 00:10:04.690915943 +0000 UTC m=+3790.578822295"
Oct 31 00:10:15.584949 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:15.584921    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855"
Oct 31 00:10:17.590656 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:10:17.590622    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:10:27.680214 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.680170    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"]
Oct 31 00:10:27.683792 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.683770    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.741030 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.740954    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f5f39e\"/\"kube-root-ca.crt\""
Oct 31 00:10:27.786234 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.786199    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f5f39e\"/\"simulator-single-main-f5f39e-kserve-self-signed-certs\""
Oct 31 00:10:27.786456 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.786234    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f5f39e\"/\"openshift-service-ca.crt\""
Oct 31 00:10:27.786456 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.786332    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f5f39e\"/\"default-dockercfg-n8m8n\""
Oct 31 00:10:27.788090 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.788071    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"]
Oct 31 00:10:27.836352 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.835972    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sztkh\" (UniqueName: \"kubernetes.io/projected/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-kube-api-access-sztkh\") pod \"simulator-single-main-f5f39e-kserve-556594964b-p77td\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") " pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.836352 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.836032    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-tls-certs\") pod \"simulator-single-main-f5f39e-kserve-556594964b-p77td\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") " pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.836352 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.836070    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-tmp-dir\") pod \"simulator-single-main-f5f39e-kserve-556594964b-p77td\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") " pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.836352 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.836107    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-home\") pod \"simulator-single-main-f5f39e-kserve-556594964b-p77td\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") " pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.836352 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.836136    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-model-cache\") pod \"simulator-single-main-f5f39e-kserve-556594964b-p77td\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") " pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.836352 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.836161    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-dshm\") pod \"simulator-single-main-f5f39e-kserve-556594964b-p77td\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") " pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.937297 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.937208    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-model-cache\") pod \"simulator-single-main-f5f39e-kserve-556594964b-p77td\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") " pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.937297 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.937251    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-dshm\") pod \"simulator-single-main-f5f39e-kserve-556594964b-p77td\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") " pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.937546 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.937403    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sztkh\" (UniqueName: \"kubernetes.io/projected/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-kube-api-access-sztkh\") pod \"simulator-single-main-f5f39e-kserve-556594964b-p77td\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") " pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.937546 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.937437    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-tls-certs\") pod \"simulator-single-main-f5f39e-kserve-556594964b-p77td\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") " pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.937546 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.937461    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-tmp-dir\") pod \"simulator-single-main-f5f39e-kserve-556594964b-p77td\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") " pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.937546 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.937496    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-home\") pod \"simulator-single-main-f5f39e-kserve-556594964b-p77td\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") " pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.937826 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.937629    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-model-cache\") pod \"simulator-single-main-f5f39e-kserve-556594964b-p77td\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") " pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.937889 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.937825    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-tmp-dir\") pod \"simulator-single-main-f5f39e-kserve-556594964b-p77td\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") " pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.938217 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.938121    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-home\") pod \"simulator-single-main-f5f39e-kserve-556594964b-p77td\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") " pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.939927 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.939908    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-dshm\") pod \"simulator-single-main-f5f39e-kserve-556594964b-p77td\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") " pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.940099 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.940083    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-tls-certs\") pod \"simulator-single-main-f5f39e-kserve-556594964b-p77td\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") " pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.994776 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.994746    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sztkh\" (UniqueName: \"kubernetes.io/projected/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-kube-api-access-sztkh\") pod \"simulator-single-main-f5f39e-kserve-556594964b-p77td\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") " pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:27.998558 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:27.998535    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:28.374672 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:28.373835    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"]
Oct 31 00:10:28.375206 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:10:28.375182    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeeb2280c_2717_4c9a_8da4_a9c3bc4f31ae.slice/crio-d8361380d30a7624d4824ff542555eddebdeda999c2783b2e745ce960b930124 WatchSource:0}: Error finding container d8361380d30a7624d4824ff542555eddebdeda999c2783b2e745ce960b930124: Status 404 returned error can't find the container with id d8361380d30a7624d4824ff542555eddebdeda999c2783b2e745ce960b930124
Oct 31 00:10:28.398781 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:28.398757    5078 patch_prober.go:29] interesting pod/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": read tcp 10.133.0.2:47860->10.133.1.28:15021: read: connection reset by peer" start-of-body=
Oct 31 00:10:28.398934 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:28.398831    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs" podUID="3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": read tcp 10.133.0.2:47860->10.133.1.28:15021: read: connection reset by peer"
Oct 31 00:10:28.724404 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:28.724280    5078 generic.go:356] "Generic (PLEG): container finished" podID="3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" containerID="b8439ad535e287b53ce813e89ed248c8c1c521862195715b668418319db871c6" exitCode=0
Oct 31 00:10:28.724404 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:28.724333    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs" event={"ID":"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d","Type":"ContainerDied","Data":"b8439ad535e287b53ce813e89ed248c8c1c521862195715b668418319db871c6"}
Oct 31 00:10:28.724939 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:28.724865    5078 scope.go:122] "RemoveContainer" containerID="b8439ad535e287b53ce813e89ed248c8c1c521862195715b668418319db871c6"
Oct 31 00:10:28.726262 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:28.726215    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:10:28.726418 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:28.726308    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:10:28.726418 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:28.726354    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:10:28.727058 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:28.727036    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td" event={"ID":"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae","Type":"ContainerStarted","Data":"bf9b8181cd55eea87e4ed180f6ed8dce58e77fefaf4e4edff7841f6e89f173d8"}
Oct 31 00:10:28.727177 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:28.727074    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td" event={"ID":"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae","Type":"ContainerStarted","Data":"d8361380d30a7624d4824ff542555eddebdeda999c2783b2e745ce960b930124"}
Oct 31 00:10:28.956185 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:28.956108    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td" podStartSLOduration=1.956088085 podStartE2EDuration="1.956088085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:28.954490538 +0000 UTC m=+3814.842396893" watchObservedRunningTime="2026-10-31 00:10:28.956088085 +0000 UTC m=+3814.843994439"
Oct 31 00:10:28.998953 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:28.998847    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:29.734000 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:29.733961    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs" event={"ID":"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d","Type":"ContainerStarted","Data":"fd71bc4e46757445f7c2a5821d040903c18e17a3cf65488fa9810b3ff2ff09a5"}
Oct 31 00:10:30.046825 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:30.046792    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:10:30.047063 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:30.047039    5078 patch_prober.go:29] interesting pod/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Oct 31 00:10:30.047113 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:30.047093    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs" podUID="3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Oct 31 00:10:31.047228 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:31.047191    5078 patch_prober.go:29] interesting pod/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Oct 31 00:10:31.047699 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:31.047259    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs" podUID="3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Oct 31 00:10:32.047120 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:32.047086    5078 patch_prober.go:29] interesting pod/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Oct 31 00:10:32.047310 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:32.047152    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs" podUID="3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Oct 31 00:10:33.046562 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:33.046528    5078 patch_prober.go:29] interesting pod/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Oct 31 00:10:33.046776 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:33.046582    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs" podUID="3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Oct 31 00:10:34.046766 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:34.046730    5078 patch_prober.go:29] interesting pod/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused" start-of-body=
Oct 31 00:10:34.047173 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:34.046782    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs" podUID="3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.28:15021/healthz/ready\": dial tcp 10.133.1.28:15021: connect: connection refused"
Oct 31 00:10:35.205034 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:35.205000    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:10:35.206076 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:35.205966    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:10:35.207356 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:35.207338    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:10:37.999165 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:37.999137    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:38.012397 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:38.012371    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:38.789715 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:10:38.789688    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:10:47.590498 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:10:47.590458    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:11:04.401083 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:04.401052    5078 scope.go:122] "RemoveContainer" containerID="84eaf4fad5cf36fb8a403b86db93933ea09ce59fa69e57cb6b3955b203fe341d"
Oct 31 00:11:04.409897 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:04.409879    5078 scope.go:122] "RemoveContainer" containerID="88232ab8b3bf06ffc9bc542d0c5b5f8058503f62e0e0bcb86bf6ba29fc127ad3"
Oct 31 00:11:04.417881 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:04.417854    5078 scope.go:122] "RemoveContainer" containerID="0bb61604f18dc737641e16f46de0a30c6edd03f108ed4ac17dc3121f56733bf8"
Oct 31 00:11:12.842734 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:12.842697    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls"]
Oct 31 00:11:12.849241 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:12.849202    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls"
Oct 31 00:11:12.903631 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:12.903567    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls"]
Oct 31 00:11:12.950731 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:12.950701    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:11:12.950901 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:12.950739    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:11:12.950901 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:12.950749    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:11:12.952383 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:12.952371    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:11:12.981691 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:12.981662    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534d2089-ffd2-46dd-9fc5-01851c312a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6e34b49-fbmls\" (UID: \"534d2089-ffd2-46dd-9fc5-01851c312a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls"
Oct 31 00:11:12.981853 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:12.981719    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534d2089-ffd2-46dd-9fc5-01851c312a6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6e34b49-fbmls\" (UID: \"534d2089-ffd2-46dd-9fc5-01851c312a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls"
Oct 31 00:11:12.981853 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:12.981804    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sh8z\" (UniqueName: \"kubernetes.io/projected/534d2089-ffd2-46dd-9fc5-01851c312a6b-kube-api-access-7sh8z\") pod \"maas-api-revoke-sub-f6e34b49-fbmls\" (UID: \"534d2089-ffd2-46dd-9fc5-01851c312a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls"
Oct 31 00:11:13.083239 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:13.083208    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534d2089-ffd2-46dd-9fc5-01851c312a6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6e34b49-fbmls\" (UID: \"534d2089-ffd2-46dd-9fc5-01851c312a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls"
Oct 31 00:11:13.083434 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:13.083250    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sh8z\" (UniqueName: \"kubernetes.io/projected/534d2089-ffd2-46dd-9fc5-01851c312a6b-kube-api-access-7sh8z\") pod \"maas-api-revoke-sub-f6e34b49-fbmls\" (UID: \"534d2089-ffd2-46dd-9fc5-01851c312a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls"
Oct 31 00:11:13.083434 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:13.083318    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534d2089-ffd2-46dd-9fc5-01851c312a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6e34b49-fbmls\" (UID: \"534d2089-ffd2-46dd-9fc5-01851c312a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls"
Oct 31 00:11:13.083979 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:13.083959    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534d2089-ffd2-46dd-9fc5-01851c312a6b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6e34b49-fbmls\" (UID: \"534d2089-ffd2-46dd-9fc5-01851c312a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls"
Oct 31 00:11:13.085913 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:13.085893    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534d2089-ffd2-46dd-9fc5-01851c312a6b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6e34b49-fbmls\" (UID: \"534d2089-ffd2-46dd-9fc5-01851c312a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls"
Oct 31 00:11:13.140638 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:13.140528    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sh8z\" (UniqueName: \"kubernetes.io/projected/534d2089-ffd2-46dd-9fc5-01851c312a6b-kube-api-access-7sh8z\") pod \"maas-api-revoke-sub-f6e34b49-fbmls\" (UID: \"534d2089-ffd2-46dd-9fc5-01851c312a6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls"
Oct 31 00:11:13.162426 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:13.162385    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls"
Oct 31 00:11:13.532210 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:13.532135    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls"]
Oct 31 00:11:13.534682 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:11:13.534655    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534d2089_ffd2_46dd_9fc5_01851c312a6b.slice/crio-b8adf4f70616f1ba2bca218ce3ce59311795d819ca9c6952667111d649436ac9 WatchSource:0}: Error finding container b8adf4f70616f1ba2bca218ce3ce59311795d819ca9c6952667111d649436ac9: Status 404 returned error can't find the container with id b8adf4f70616f1ba2bca218ce3ce59311795d819ca9c6952667111d649436ac9
Oct 31 00:11:13.948091 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:13.948056    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls" event={"ID":"534d2089-ffd2-46dd-9fc5-01851c312a6b","Type":"ContainerStarted","Data":"4dc0b376e2af9fd44da77da2a4adc084656c45f2619fd36ceafeec1e22613479"}
Oct 31 00:11:13.948642 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:13.948119    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls" event={"ID":"534d2089-ffd2-46dd-9fc5-01851c312a6b","Type":"ContainerStarted","Data":"b8adf4f70616f1ba2bca218ce3ce59311795d819ca9c6952667111d649436ac9"}
Oct 31 00:11:14.954758 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:14.954722    5078 generic.go:356] "Generic (PLEG): container finished" podID="534d2089-ffd2-46dd-9fc5-01851c312a6b" containerID="4dc0b376e2af9fd44da77da2a4adc084656c45f2619fd36ceafeec1e22613479" exitCode=0
Oct 31 00:11:14.955167 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:14.954802    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls" event={"ID":"534d2089-ffd2-46dd-9fc5-01851c312a6b","Type":"ContainerDied","Data":"4dc0b376e2af9fd44da77da2a4adc084656c45f2619fd36ceafeec1e22613479"}
Oct 31 00:11:16.093517 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:16.093495    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls"
Oct 31 00:11:16.212334 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:16.212245    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/534d2089-ffd2-46dd-9fc5-01851c312a6b-kube-api-access-7sh8z\" (UniqueName: \"kubernetes.io/projected/534d2089-ffd2-46dd-9fc5-01851c312a6b-kube-api-access-7sh8z\") pod \"534d2089-ffd2-46dd-9fc5-01851c312a6b\" (UID: \"534d2089-ffd2-46dd-9fc5-01851c312a6b\") "
Oct 31 00:11:16.212522 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:16.212333    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/534d2089-ffd2-46dd-9fc5-01851c312a6b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534d2089-ffd2-46dd-9fc5-01851c312a6b-maas-api-xks-ca\") pod \"534d2089-ffd2-46dd-9fc5-01851c312a6b\" (UID: \"534d2089-ffd2-46dd-9fc5-01851c312a6b\") "
Oct 31 00:11:16.212522 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:16.212383    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/534d2089-ffd2-46dd-9fc5-01851c312a6b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534d2089-ffd2-46dd-9fc5-01851c312a6b-maas-api-openshift-ca\") pod \"534d2089-ffd2-46dd-9fc5-01851c312a6b\" (UID: \"534d2089-ffd2-46dd-9fc5-01851c312a6b\") "
Oct 31 00:11:16.212794 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:16.212767    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/534d2089-ffd2-46dd-9fc5-01851c312a6b-maas-api-openshift-ca" pod "534d2089-ffd2-46dd-9fc5-01851c312a6b" (UID: "534d2089-ffd2-46dd-9fc5-01851c312a6b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:11:16.214638 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:16.214612    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/534d2089-ffd2-46dd-9fc5-01851c312a6b-maas-api-xks-ca" pod "534d2089-ffd2-46dd-9fc5-01851c312a6b" (UID: "534d2089-ffd2-46dd-9fc5-01851c312a6b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:11:16.215233 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:16.215209    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/534d2089-ffd2-46dd-9fc5-01851c312a6b-kube-api-access-7sh8z" pod "534d2089-ffd2-46dd-9fc5-01851c312a6b" (UID: "534d2089-ffd2-46dd-9fc5-01851c312a6b"). InnerVolumeSpecName "kube-api-access-7sh8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:11:16.313767 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:16.313740    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sh8z\" (UniqueName: \"kubernetes.io/projected/534d2089-ffd2-46dd-9fc5-01851c312a6b-kube-api-access-7sh8z\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:16.313767 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:16.313767    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/534d2089-ffd2-46dd-9fc5-01851c312a6b-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:16.313956 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:16.313778    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/534d2089-ffd2-46dd-9fc5-01851c312a6b-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:16.966392 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:16.966360    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls"
Oct 31 00:11:16.966392 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:16.966385    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls" event={"ID":"534d2089-ffd2-46dd-9fc5-01851c312a6b","Type":"ContainerDied","Data":"b8adf4f70616f1ba2bca218ce3ce59311795d819ca9c6952667111d649436ac9"}
Oct 31 00:11:16.966648 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:16.966452    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8adf4f70616f1ba2bca218ce3ce59311795d819ca9c6952667111d649436ac9"
Oct 31 00:11:17.590673 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:11:17.590635    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:11:24.407503 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.407458    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk"]
Oct 31 00:11:24.408038 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.407866    5078 cpu_manager.go:414] "containerMap: removing container" podUID="534d2089-ffd2-46dd-9fc5-01851c312a6b" containerName="revoke-keys"
Oct 31 00:11:24.408038 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.407879    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="534d2089-ffd2-46dd-9fc5-01851c312a6b" containerName="revoke-keys"
Oct 31 00:11:24.408038 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.407947    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="534d2089-ffd2-46dd-9fc5-01851c312a6b" containerName="revoke-keys"
Oct 31 00:11:24.410899 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.410882    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk"
Oct 31 00:11:24.483035 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.483001    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85461e91-3d40-4989-bf69-8892c7a1aa48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk\" (UID: \"85461e91-3d40-4989-bf69-8892c7a1aa48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk"
Oct 31 00:11:24.483200 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.483042    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqczq\" (UniqueName: \"kubernetes.io/projected/85461e91-3d40-4989-bf69-8892c7a1aa48-kube-api-access-qqczq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk\" (UID: \"85461e91-3d40-4989-bf69-8892c7a1aa48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk"
Oct 31 00:11:24.483200 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.483153    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85461e91-3d40-4989-bf69-8892c7a1aa48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk\" (UID: \"85461e91-3d40-4989-bf69-8892c7a1aa48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk"
Oct 31 00:11:24.514411 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.514314    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:11:24.514411 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.514355    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:11:24.514411 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.514401    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:11:24.514733 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.514420    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:11:24.516254 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.516230    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk"]
Oct 31 00:11:24.583567 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.583527    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85461e91-3d40-4989-bf69-8892c7a1aa48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk\" (UID: \"85461e91-3d40-4989-bf69-8892c7a1aa48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk"
Oct 31 00:11:24.583746 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.583573    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqczq\" (UniqueName: \"kubernetes.io/projected/85461e91-3d40-4989-bf69-8892c7a1aa48-kube-api-access-qqczq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk\" (UID: \"85461e91-3d40-4989-bf69-8892c7a1aa48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk"
Oct 31 00:11:24.583746 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.583720    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85461e91-3d40-4989-bf69-8892c7a1aa48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk\" (UID: \"85461e91-3d40-4989-bf69-8892c7a1aa48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk"
Oct 31 00:11:24.584430 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.584346    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85461e91-3d40-4989-bf69-8892c7a1aa48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk\" (UID: \"85461e91-3d40-4989-bf69-8892c7a1aa48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk"
Oct 31 00:11:24.586069 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.586049    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85461e91-3d40-4989-bf69-8892c7a1aa48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk\" (UID: \"85461e91-3d40-4989-bf69-8892c7a1aa48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk"
Oct 31 00:11:24.641141 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.641105    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqczq\" (UniqueName: \"kubernetes.io/projected/85461e91-3d40-4989-bf69-8892c7a1aa48-kube-api-access-qqczq\") pod \"maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk\" (UID: \"85461e91-3d40-4989-bf69-8892c7a1aa48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk"
Oct 31 00:11:24.723953 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:24.723872    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk"
Oct 31 00:11:25.091968 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:25.091948    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk"]
Oct 31 00:11:25.094300 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:11:25.094275    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85461e91_3d40_4989_bf69_8892c7a1aa48.slice/crio-0af6f79abd1941216b771661e82c309750dcaff6b2a911df90c55d47031ed81e WatchSource:0}: Error finding container 0af6f79abd1941216b771661e82c309750dcaff6b2a911df90c55d47031ed81e: Status 404 returned error can't find the container with id 0af6f79abd1941216b771661e82c309750dcaff6b2a911df90c55d47031ed81e
Oct 31 00:11:26.010445 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:26.010400    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk" event={"ID":"85461e91-3d40-4989-bf69-8892c7a1aa48","Type":"ContainerStarted","Data":"84a46391484248f7507218ae917a3b11524fb1ba31873fbc5292c0fbcda6b5fd"}
Oct 31 00:11:26.011021 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:26.010466    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk" event={"ID":"85461e91-3d40-4989-bf69-8892c7a1aa48","Type":"ContainerStarted","Data":"0af6f79abd1941216b771661e82c309750dcaff6b2a911df90c55d47031ed81e"}
Oct 31 00:11:26.123919 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:26.123861    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk" podStartSLOduration=2.123843267 podStartE2EDuration="2.123843267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:26.12311194 +0000 UTC m=+3872.011018293" watchObservedRunningTime="2026-10-31 00:11:26.123843267 +0000 UTC m=+3872.011749622"
Oct 31 00:11:27.016274 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:27.016237    5078 generic.go:356] "Generic (PLEG): container finished" podID="85461e91-3d40-4989-bf69-8892c7a1aa48" containerID="84a46391484248f7507218ae917a3b11524fb1ba31873fbc5292c0fbcda6b5fd" exitCode=0
Oct 31 00:11:27.016725 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:27.016298    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk" event={"ID":"85461e91-3d40-4989-bf69-8892c7a1aa48","Type":"ContainerDied","Data":"84a46391484248f7507218ae917a3b11524fb1ba31873fbc5292c0fbcda6b5fd"}
Oct 31 00:11:28.153159 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:28.153130    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk"
Oct 31 00:11:28.216854 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:28.216818    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85461e91-3d40-4989-bf69-8892c7a1aa48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85461e91-3d40-4989-bf69-8892c7a1aa48-maas-api-xks-ca\") pod \"85461e91-3d40-4989-bf69-8892c7a1aa48\" (UID: \"85461e91-3d40-4989-bf69-8892c7a1aa48\") "
Oct 31 00:11:28.217032 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:28.216866    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85461e91-3d40-4989-bf69-8892c7a1aa48-kube-api-access-qqczq\" (UniqueName: \"kubernetes.io/projected/85461e91-3d40-4989-bf69-8892c7a1aa48-kube-api-access-qqczq\") pod \"85461e91-3d40-4989-bf69-8892c7a1aa48\" (UID: \"85461e91-3d40-4989-bf69-8892c7a1aa48\") "
Oct 31 00:11:28.217032 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:28.216985    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85461e91-3d40-4989-bf69-8892c7a1aa48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85461e91-3d40-4989-bf69-8892c7a1aa48-maas-api-openshift-ca\") pod \"85461e91-3d40-4989-bf69-8892c7a1aa48\" (UID: \"85461e91-3d40-4989-bf69-8892c7a1aa48\") "
Oct 31 00:11:28.217336 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:28.217311    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85461e91-3d40-4989-bf69-8892c7a1aa48-maas-api-openshift-ca" pod "85461e91-3d40-4989-bf69-8892c7a1aa48" (UID: "85461e91-3d40-4989-bf69-8892c7a1aa48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:11:28.219035 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:28.219011    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85461e91-3d40-4989-bf69-8892c7a1aa48-maas-api-xks-ca" pod "85461e91-3d40-4989-bf69-8892c7a1aa48" (UID: "85461e91-3d40-4989-bf69-8892c7a1aa48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:11:28.219140 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:28.219056    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85461e91-3d40-4989-bf69-8892c7a1aa48-kube-api-access-qqczq" pod "85461e91-3d40-4989-bf69-8892c7a1aa48" (UID: "85461e91-3d40-4989-bf69-8892c7a1aa48"). InnerVolumeSpecName "kube-api-access-qqczq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:11:28.318120 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:28.318092    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/85461e91-3d40-4989-bf69-8892c7a1aa48-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:28.318120 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:28.318118    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqczq\" (UniqueName: \"kubernetes.io/projected/85461e91-3d40-4989-bf69-8892c7a1aa48-kube-api-access-qqczq\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:28.318302 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:28.318129    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/85461e91-3d40-4989-bf69-8892c7a1aa48-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:28.892023 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:28.891987    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"]
Oct 31 00:11:29.039142 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:29.039103    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk" event={"ID":"85461e91-3d40-4989-bf69-8892c7a1aa48","Type":"ContainerDied","Data":"0af6f79abd1941216b771661e82c309750dcaff6b2a911df90c55d47031ed81e"}
Oct 31 00:11:29.039320 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:29.039158    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0af6f79abd1941216b771661e82c309750dcaff6b2a911df90c55d47031ed81e"
Oct 31 00:11:29.039320 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:29.039203    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk"
Oct 31 00:11:36.010376 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:36.010289    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855"]
Oct 31 00:11:41.011185 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:41.011123    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855" podUID="8a0a9651-9b67-4834-a2e6-01d713044774" containerName="payload-pre-processing" containerID="cri-o://983d6dddc4731e6f4e4c2cdf6153a92d217065fdb16e9c2a7ade461455a6b43b" gracePeriod=25
Oct 31 00:11:43.912306 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:43.912254    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td" podUID="eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae" containerName="main" containerID="cri-o://bf9b8181cd55eea87e4ed180f6ed8dce58e77fefaf4e4edff7841f6e89f173d8" gracePeriod=45
Oct 31 00:11:44.106676 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.106647    5078 generic.go:356] "Generic (PLEG): container finished" podID="eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae" containerID="bf9b8181cd55eea87e4ed180f6ed8dce58e77fefaf4e4edff7841f6e89f173d8" exitCode=0
Oct 31 00:11:44.106823 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.106704    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td" event={"ID":"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae","Type":"ContainerDied","Data":"bf9b8181cd55eea87e4ed180f6ed8dce58e77fefaf4e4edff7841f6e89f173d8"}
Oct 31 00:11:44.163327 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.163271    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:11:44.280207 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.280171    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-dshm\") pod \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") "
Oct 31 00:11:44.280399 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.280264    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-model-cache\") pod \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") "
Oct 31 00:11:44.280399 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.280343    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-tmp-dir\") pod \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") "
Oct 31 00:11:44.280535 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.280397    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-home\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-home\") pod \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") "
Oct 31 00:11:44.280535 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.280431    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-tls-certs\") pod \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") "
Oct 31 00:11:44.280535 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.280460    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-kube-api-access-sztkh\" (UniqueName: \"kubernetes.io/projected/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-kube-api-access-sztkh\") pod \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\" (UID: \"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae\") "
Oct 31 00:11:44.280723 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.280646    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-tmp-dir" pod "eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae" (UID: "eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:44.280723 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.280694    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-home" pod "eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae" (UID: "eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:44.280838 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.280822    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:44.280897 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.280844    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:44.281073 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.281054    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-model-cache" pod "eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae" (UID: "eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:44.282930 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.282896    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-tls-certs" pod "eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae" (UID: "eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:11:44.285037 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.283430    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-dshm" pod "eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae" (UID: "eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:44.290627 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.290535    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-kube-api-access-sztkh" pod "eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae" (UID: "eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae"). InnerVolumeSpecName "kube-api-access-sztkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:11:44.381808 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.381778    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:44.381808 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.381806    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:44.381808 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.381817    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sztkh\" (UniqueName: \"kubernetes.io/projected/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-kube-api-access-sztkh\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:44.382058 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.381826    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:44.786825 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.786792    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk"]
Oct 31 00:11:44.789586 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:44.789561    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-f5f39e-b77qk"]
Oct 31 00:11:45.113371 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:45.113340    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"
Oct 31 00:11:45.113371 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:45.113369    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td" event={"ID":"eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae","Type":"ContainerDied","Data":"d8361380d30a7624d4824ff542555eddebdeda999c2783b2e745ce960b930124"}
Oct 31 00:11:45.113928 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:45.113413    5078 scope.go:122] "RemoveContainer" containerID="bf9b8181cd55eea87e4ed180f6ed8dce58e77fefaf4e4edff7841f6e89f173d8"
Oct 31 00:11:45.277097 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:45.277066    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"]
Oct 31 00:11:45.326469 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:45.326442    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-f5f39e/simulator-single-main-f5f39e-kserve-556594964b-p77td"]
Oct 31 00:11:45.583340 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:45.583303    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855" podUID="8a0a9651-9b67-4834-a2e6-01d713044774" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:11:46.050286 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:46.050252    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn"]
Oct 31 00:11:46.102924 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:46.101472    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01fa745c-gccqn"]
Oct 31 00:11:46.296939 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:46.296911    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85461e91-3d40-4989-bf69-8892c7a1aa48" path="/var/lib/kubelet/pods/85461e91-3d40-4989-bf69-8892c7a1aa48/volumes"
Oct 31 00:11:46.297303 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:46.297286    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae" path="/var/lib/kubelet/pods/eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae/volumes"
Oct 31 00:11:46.297670 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:46.297656    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5831912-d0b7-4878-9187-560c8f9627a4" path="/var/lib/kubelet/pods/f5831912-d0b7-4878-9187-560c8f9627a4/volumes"
Oct 31 00:11:47.589821 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:11:47.589787    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:11:51.791713 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:51.791678    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"]
Oct 31 00:11:51.792282 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:51.791961    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs" podUID="3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" containerName="istio-proxy" containerID="cri-o://fd71bc4e46757445f7c2a5821d040903c18e17a3cf65488fa9810b3ff2ff09a5" gracePeriod=30
Oct 31 00:11:55.583844 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:55.583800    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855" podUID="8a0a9651-9b67-4834-a2e6-01d713044774" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:11:57.051479 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.051453    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:11:57.169796 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.169708    5078 generic.go:356] "Generic (PLEG): container finished" podID="3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" containerID="fd71bc4e46757445f7c2a5821d040903c18e17a3cf65488fa9810b3ff2ff09a5" exitCode=0
Oct 31 00:11:57.169796 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.169761    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"
Oct 31 00:11:57.169796 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.169774    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs" event={"ID":"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d","Type":"ContainerDied","Data":"fd71bc4e46757445f7c2a5821d040903c18e17a3cf65488fa9810b3ff2ff09a5"}
Oct 31 00:11:57.170049 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.169835    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs" event={"ID":"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d","Type":"ContainerDied","Data":"ffb73197318a051f1ad7cdac8bbf965f69d5a8816c4cfb079662f4bb1caba59e"}
Oct 31 00:11:57.170049 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.169856    5078 scope.go:122] "RemoveContainer" containerID="fd71bc4e46757445f7c2a5821d040903c18e17a3cf65488fa9810b3ff2ff09a5"
Oct 31 00:11:57.178190 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.178173    5078 scope.go:122] "RemoveContainer" containerID="b8439ad535e287b53ce813e89ed248c8c1c521862195715b668418319db871c6"
Oct 31 00:11:57.186028 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.186006    5078 scope.go:122] "RemoveContainer" containerID="fd71bc4e46757445f7c2a5821d040903c18e17a3cf65488fa9810b3ff2ff09a5"
Oct 31 00:11:57.186305 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:11:57.186286    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd71bc4e46757445f7c2a5821d040903c18e17a3cf65488fa9810b3ff2ff09a5\": container with ID starting with fd71bc4e46757445f7c2a5821d040903c18e17a3cf65488fa9810b3ff2ff09a5 not found: ID does not exist" containerID="fd71bc4e46757445f7c2a5821d040903c18e17a3cf65488fa9810b3ff2ff09a5"
Oct 31 00:11:57.186441 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.186316    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd71bc4e46757445f7c2a5821d040903c18e17a3cf65488fa9810b3ff2ff09a5"} err="failed to get container status \"fd71bc4e46757445f7c2a5821d040903c18e17a3cf65488fa9810b3ff2ff09a5\": rpc error: code = NotFound desc = could not find container \"fd71bc4e46757445f7c2a5821d040903c18e17a3cf65488fa9810b3ff2ff09a5\": container with ID starting with fd71bc4e46757445f7c2a5821d040903c18e17a3cf65488fa9810b3ff2ff09a5 not found: ID does not exist"
Oct 31 00:11:57.186441 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.186334    5078 scope.go:122] "RemoveContainer" containerID="b8439ad535e287b53ce813e89ed248c8c1c521862195715b668418319db871c6"
Oct 31 00:11:57.186615 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:11:57.186575    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8439ad535e287b53ce813e89ed248c8c1c521862195715b668418319db871c6\": container with ID starting with b8439ad535e287b53ce813e89ed248c8c1c521862195715b668418319db871c6 not found: ID does not exist" containerID="b8439ad535e287b53ce813e89ed248c8c1c521862195715b668418319db871c6"
Oct 31 00:11:57.186670 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.186615    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8439ad535e287b53ce813e89ed248c8c1c521862195715b668418319db871c6"} err="failed to get container status \"b8439ad535e287b53ce813e89ed248c8c1c521862195715b668418319db871c6\": rpc error: code = NotFound desc = could not find container \"b8439ad535e287b53ce813e89ed248c8c1c521862195715b668418319db871c6\": container with ID starting with b8439ad535e287b53ce813e89ed248c8c1c521862195715b668418319db871c6 not found: ID does not exist"
Oct 31 00:11:57.207820 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.207800    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-workload-certs\") pod \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") "
Oct 31 00:11:57.207908 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.207831    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-workload-socket\") pod \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") "
Oct 31 00:11:57.207908 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.207873    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-data\") pod \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") "
Oct 31 00:11:57.207908 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.207896    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-credential-socket\") pod \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") "
Oct 31 00:11:57.208055 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.207925    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-envoy\") pod \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") "
Oct 31 00:11:57.208055 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.207947    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istiod-ca-cert\") pod \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") "
Oct 31 00:11:57.208055 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.208002    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-token\" (UniqueName: \"kubernetes.io/projected/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-token\") pod \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") "
Oct 31 00:11:57.208215 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.208105    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-podinfo\") pod \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") "
Oct 31 00:11:57.208215 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.208091    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-workload-certs" pod "3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" (UID: "3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:57.208215 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.208143    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-kube-api-access-gmhqs\" (UniqueName: \"kubernetes.io/projected/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-kube-api-access-gmhqs\") pod \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\" (UID: \"3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d\") "
Oct 31 00:11:57.208215 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.208128    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-workload-socket" pod "3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" (UID: "3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:57.208215 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.208164    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-credential-socket" pod "3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" (UID: "3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:57.208215 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.208191    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-data" pod "3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" (UID: "3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:57.208511 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.208381    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istiod-ca-cert" pod "3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" (UID: "3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:11:57.208555 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.208515    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:57.208555 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.208539    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:57.208681 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.208556    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:57.208681 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.208571    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:57.208681 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.208584    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:57.210250 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.210228    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-token" pod "3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" (UID: "3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:11:57.210373 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.210352    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-kube-api-access-gmhqs" pod "3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" (UID: "3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d"). InnerVolumeSpecName "kube-api-access-gmhqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:11:57.210495 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.210473    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-podinfo" pod "3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" (UID: "3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:11:57.210553 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.210496    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-envoy" pod "3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" (UID: "3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:11:57.309934 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.309900    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:57.309934 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.309931    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:57.309934 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.309942    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:57.310185 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.309954    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmhqs\" (UniqueName: \"kubernetes.io/projected/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d-kube-api-access-gmhqs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:57.589859 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.589826    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"]
Oct 31 00:11:57.595898 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:57.595872    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f5f39e-openshift-default-755bfbfbcchpzvs"]
Oct 31 00:11:58.297113 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:58.297072    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" path="/var/lib/kubelet/pods/3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d/volumes"
Oct 31 00:11:59.363052 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.363018    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"]
Oct 31 00:11:59.363463 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.363333    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" containerName="istio-proxy"
Oct 31 00:11:59.363463 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.363343    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" containerName="istio-proxy"
Oct 31 00:11:59.363463 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.363370    5078 cpu_manager.go:414] "containerMap: removing container" podUID="eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae" containerName="main"
Oct 31 00:11:59.363463 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.363375    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae" containerName="main"
Oct 31 00:11:59.363463 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.363384    5078 cpu_manager.go:414] "containerMap: removing container" podUID="85461e91-3d40-4989-bf69-8892c7a1aa48" containerName="revoke-keys"
Oct 31 00:11:59.363463 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.363388    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85461e91-3d40-4989-bf69-8892c7a1aa48" containerName="revoke-keys"
Oct 31 00:11:59.363463 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.363439    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb2280c-2717-4c9a-8da4-a9c3bc4f31ae" containerName="main"
Oct 31 00:11:59.363463 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.363446    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" containerName="istio-proxy"
Oct 31 00:11:59.363463 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.363452    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="85461e91-3d40-4989-bf69-8892c7a1aa48" containerName="revoke-keys"
Oct 31 00:11:59.363463 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.363460    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" containerName="istio-proxy"
Oct 31 00:11:59.366505 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.366489    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.470768 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.470736    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-ff863c73-kserve-self-signed-certs\""
Oct 31 00:11:59.472355 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.472330    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"]
Oct 31 00:11:59.532828 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.532792    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-home\") pod \"e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") " pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.533145 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.533116    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmmvq\" (UniqueName: \"kubernetes.io/projected/6b2368ac-55f6-496b-978c-3feed0d833fb-kube-api-access-zmmvq\") pod \"e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") " pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.533350 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.533169    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-dshm\") pod \"e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") " pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.533442 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.533343    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-model-cache\") pod \"e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") " pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.533442 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.533383    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2368ac-55f6-496b-978c-3feed0d833fb-tls-certs\") pod \"e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") " pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.533442 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.533423    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-tmp-dir\") pod \"e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") " pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.634422 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.634327    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-model-cache\") pod \"e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") " pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.634422 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.634366    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2368ac-55f6-496b-978c-3feed0d833fb-tls-certs\") pod \"e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") " pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.634736 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.634490    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-tmp-dir\") pod \"e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") " pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.634736 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.634550    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-home\") pod \"e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") " pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.634736 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.634578    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmmvq\" (UniqueName: \"kubernetes.io/projected/6b2368ac-55f6-496b-978c-3feed0d833fb-kube-api-access-zmmvq\") pod \"e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") " pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.634736 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.634638    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-dshm\") pod \"e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") " pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.634950 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.634790    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-model-cache\") pod \"e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") " pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.634950 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.634842    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-tmp-dir\") pod \"e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") " pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.634950 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.634879    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-home\") pod \"e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") " pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.636980 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.636955    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-dshm\") pod \"e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") " pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.637125 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.637107    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2368ac-55f6-496b-978c-3feed0d833fb-tls-certs\") pod \"e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") " pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.691680 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.691649    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmmvq\" (UniqueName: \"kubernetes.io/projected/6b2368ac-55f6-496b-978c-3feed0d833fb-kube-api-access-zmmvq\") pod \"e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") " pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:11:59.977582 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:11:59.977475    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:12:00.349334 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:00.349310    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"]
Oct 31 00:12:00.352225 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:12:00.352195    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b2368ac_55f6_496b_978c_3feed0d833fb.slice/crio-02837588bd955529f59e6ffd1665c7808699f6782f30016496491601289e1c9c WatchSource:0}: Error finding container 02837588bd955529f59e6ffd1665c7808699f6782f30016496491601289e1c9c: Status 404 returned error can't find the container with id 02837588bd955529f59e6ffd1665c7808699f6782f30016496491601289e1c9c
Oct 31 00:12:01.191013 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:01.190984    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6" event={"ID":"6b2368ac-55f6-496b-978c-3feed0d833fb","Type":"ContainerStarted","Data":"af9b845ac5bf1697d6073ca5a5a209c0321a77ff9c736ed0716e728f15f50680"}
Oct 31 00:12:01.191469 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:01.191030    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6" event={"ID":"6b2368ac-55f6-496b-978c-3feed0d833fb","Type":"ContainerStarted","Data":"02837588bd955529f59e6ffd1665c7808699f6782f30016496491601289e1c9c"}
Oct 31 00:12:01.192565 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:01.192543    5078 generic.go:356] "Generic (PLEG): container finished" podID="8a0a9651-9b67-4834-a2e6-01d713044774" containerID="983d6dddc4731e6f4e4c2cdf6153a92d217065fdb16e9c2a7ade461455a6b43b" exitCode=0
Oct 31 00:12:01.192679 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:01.192575    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855" event={"ID":"8a0a9651-9b67-4834-a2e6-01d713044774","Type":"ContainerDied","Data":"983d6dddc4731e6f4e4c2cdf6153a92d217065fdb16e9c2a7ade461455a6b43b"}
Oct 31 00:12:01.252855 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:01.252823    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855"
Oct 31 00:12:01.305251 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:01.305197    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6" podStartSLOduration=2.30518145 podStartE2EDuration="2.30518145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:01.304467797 +0000 UTC m=+3907.192374151" watchObservedRunningTime="2026-10-31 00:12:01.30518145 +0000 UTC m=+3907.193087803"
Oct 31 00:12:01.349991 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:01.349957    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a0a9651-9b67-4834-a2e6-01d713044774-kube-api-access-b75j4\" (UniqueName: \"kubernetes.io/projected/8a0a9651-9b67-4834-a2e6-01d713044774-kube-api-access-b75j4\") pod \"8a0a9651-9b67-4834-a2e6-01d713044774\" (UID: \"8a0a9651-9b67-4834-a2e6-01d713044774\") "
Oct 31 00:12:01.350190 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:01.350000    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a0a9651-9b67-4834-a2e6-01d713044774-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a0a9651-9b67-4834-a2e6-01d713044774-plugins-config-volume\") pod \"8a0a9651-9b67-4834-a2e6-01d713044774\" (UID: \"8a0a9651-9b67-4834-a2e6-01d713044774\") "
Oct 31 00:12:01.350433 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:01.350406    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a0a9651-9b67-4834-a2e6-01d713044774-plugins-config-volume" pod "8a0a9651-9b67-4834-a2e6-01d713044774" (UID: "8a0a9651-9b67-4834-a2e6-01d713044774"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:12:01.352228 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:01.352210    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a0a9651-9b67-4834-a2e6-01d713044774-kube-api-access-b75j4" pod "8a0a9651-9b67-4834-a2e6-01d713044774" (UID: "8a0a9651-9b67-4834-a2e6-01d713044774"). InnerVolumeSpecName "kube-api-access-b75j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:12:01.451088 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:01.451057    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b75j4\" (UniqueName: \"kubernetes.io/projected/8a0a9651-9b67-4834-a2e6-01d713044774-kube-api-access-b75j4\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:12:01.451088 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:01.451086    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a0a9651-9b67-4834-a2e6-01d713044774-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:12:01.977695 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:01.977660    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:12:02.198843 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:02.198814    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855"
Oct 31 00:12:02.199262 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:02.198844    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855" event={"ID":"8a0a9651-9b67-4834-a2e6-01d713044774","Type":"ContainerDied","Data":"43108dee820a5a046ef0ce67b2311b3e48e866d8cbabc735eb6bc1ecf6e9f1fa"}
Oct 31 00:12:02.199262 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:02.198901    5078 scope.go:122] "RemoveContainer" containerID="983d6dddc4731e6f4e4c2cdf6153a92d217065fdb16e9c2a7ade461455a6b43b"
Oct 31 00:12:02.360462 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:02.360427    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855"]
Oct 31 00:12:02.410043 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:02.410005    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f5f39e-7d978rl855"]
Oct 31 00:12:04.297566 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:04.297535    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a0a9651-9b67-4834-a2e6-01d713044774" path="/var/lib/kubelet/pods/8a0a9651-9b67-4834-a2e6-01d713044774/volumes"
Oct 31 00:12:04.461644 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:04.461614    5078 scope.go:122] "RemoveContainer" containerID="4331cacfc410be11248002cf55c817647efb7c3fa3ddb61d56f4f08e5f28b59e"
Oct 31 00:12:06.733883 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:06.733829    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-42wvb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:12:06.734487 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:06.734456    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-42wvb" podUID="c76a2fb3-5cc9-4e1d-bc93-c8acf8723819" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:12:06.743854 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:06.743733    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4662g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:12:06.743854 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:06.743791    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4662g" podUID="9a60053e-3454-4ef1-9b40-9194f19b840d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.2:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:12:07.266623 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:07.257781    5078 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4g4vj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:12:07.266623 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:07.257860    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4g4vj" podUID="9af70286-d197-473c-be7d-9333a83d1c4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.7:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:12:09.978249 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:09.978213    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:12:09.990490 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:09.990464    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:12:10.287045 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:10.286960    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:12:17.589753 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:12:17.589717    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:12:27.044883 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:27.044851    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7"]
Oct 31 00:12:27.094397 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:27.094358    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9mnq7"]
Oct 31 00:12:28.296687 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:28.296649    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371a9eb4-51eb-4d3d-b1f7-462d0321828b" path="/var/lib/kubelet/pods/371a9eb4-51eb-4d3d-b1f7-462d0321828b/volumes"
Oct 31 00:12:42.224441 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.224401    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"]
Oct 31 00:12:42.224994 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.224947    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" containerName="istio-proxy"
Oct 31 00:12:42.224994 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.224967    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f84e47a-b5d9-4a4e-a2d3-fc3c5b46318d" containerName="istio-proxy"
Oct 31 00:12:42.224994 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.224984    5078 cpu_manager.go:414] "containerMap: removing container" podUID="8a0a9651-9b67-4834-a2e6-01d713044774" containerName="payload-pre-processing"
Oct 31 00:12:42.224994 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.224992    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a0a9651-9b67-4834-a2e6-01d713044774" containerName="payload-pre-processing"
Oct 31 00:12:42.225291 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.225075    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a0a9651-9b67-4834-a2e6-01d713044774" containerName="payload-pre-processing"
Oct 31 00:12:42.228329 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.228308    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.285204 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.285168    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"]
Oct 31 00:12:42.332675 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.332648    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-ff863c73-kserve-self-signed-certs\""
Oct 31 00:12:42.371140 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.371102    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-home\") pod \"e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") " pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.371357 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.371168    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-dshm\") pod \"e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") " pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.371357 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.371201    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjctp\" (UniqueName: \"kubernetes.io/projected/d4045f91-56ef-4295-a487-a4ffb741d831-kube-api-access-kjctp\") pod \"e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") " pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.371357 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.371262    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-tmp-dir\") pod \"e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") " pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.371357 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.371321    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4045f91-56ef-4295-a487-a4ffb741d831-tls-certs\") pod \"e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") " pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.372327 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.371375    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-model-cache\") pod \"e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") " pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.472284 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.472257    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-home\") pod \"e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") " pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.472482 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.472296    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-dshm\") pod \"e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") " pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.472482 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.472317    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjctp\" (UniqueName: \"kubernetes.io/projected/d4045f91-56ef-4295-a487-a4ffb741d831-kube-api-access-kjctp\") pod \"e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") " pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.472482 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.472348    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-tmp-dir\") pod \"e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") " pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.472482 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.472374    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4045f91-56ef-4295-a487-a4ffb741d831-tls-certs\") pod \"e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") " pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.472482 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.472403    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-model-cache\") pod \"e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") " pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.472824 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.472760    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-home\") pod \"e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") " pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.472824 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.472794    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-tmp-dir\") pod \"e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") " pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.472824 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.472815    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-model-cache\") pod \"e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") " pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.474793 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.474736    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-dshm\") pod \"e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") " pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.475023 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.475003    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4045f91-56ef-4295-a487-a4ffb741d831-tls-certs\") pod \"e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") " pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.530183 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.530153    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjctp\" (UniqueName: \"kubernetes.io/projected/d4045f91-56ef-4295-a487-a4ffb741d831-kube-api-access-kjctp\") pod \"e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") " pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.539474 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.539437    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:42.916394 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:42.916366    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"]
Oct 31 00:12:42.918577 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:12:42.918548    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4045f91_56ef_4295_a487_a4ffb741d831.slice/crio-15b9872ea974b8b246a0aa6b9c7d95917ec53c68eddccba0252ad3052ef65627 WatchSource:0}: Error finding container 15b9872ea974b8b246a0aa6b9c7d95917ec53c68eddccba0252ad3052ef65627: Status 404 returned error can't find the container with id 15b9872ea974b8b246a0aa6b9c7d95917ec53c68eddccba0252ad3052ef65627
Oct 31 00:12:43.420520 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:43.420489    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9" event={"ID":"d4045f91-56ef-4295-a487-a4ffb741d831","Type":"ContainerStarted","Data":"fabe2e63b72bb26901dfa1c0dc69ce18cdc412193066d7a477cbb55697bc1ac9"}
Oct 31 00:12:43.420520 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:43.420527    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9" event={"ID":"d4045f91-56ef-4295-a487-a4ffb741d831","Type":"ContainerStarted","Data":"15b9872ea974b8b246a0aa6b9c7d95917ec53c68eddccba0252ad3052ef65627"}
Oct 31 00:12:43.534247 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:43.534194    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9" podStartSLOduration=1.534178254 podStartE2EDuration="1.534178254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:43.532949624 +0000 UTC m=+3949.420855978" watchObservedRunningTime="2026-10-31 00:12:43.534178254 +0000 UTC m=+3949.422084607"
Oct 31 00:12:43.539740 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:43.539715    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:47.710082 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:12:47.710011    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:12:52.540281 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:52.540247    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:52.553032 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:52.553005    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:12:53.488335 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:12:53.488307    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:13:04.496565 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:04.496488    5078 scope.go:122] "RemoveContainer" containerID="70e9d987f60a0624799b713756687ed35b1be562c6acd2af585493972b0d0668"
Oct 31 00:13:04.505906 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:04.505882    5078 scope.go:122] "RemoveContainer" containerID="2255a6fd0f642ed0a3603d26fb27c011ee7b2063185ba8905eee4f8ad2c017e2"
Oct 31 00:13:13.340242 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:13.340207    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts"]
Oct 31 00:13:13.344973 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:13.344953    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts"
Oct 31 00:13:13.398739 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:13.398262    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cbb295-c661-43ea-bc33-8cbae904fa7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-rszts\" (UID: \"62cbb295-c661-43ea-bc33-8cbae904fa7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts"
Oct 31 00:13:13.398739 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:13.398345    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8gw2\" (UniqueName: \"kubernetes.io/projected/62cbb295-c661-43ea-bc33-8cbae904fa7e-kube-api-access-h8gw2\") pod \"maas-api-revoke-sub-452b6d7b-rszts\" (UID: \"62cbb295-c661-43ea-bc33-8cbae904fa7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts"
Oct 31 00:13:13.398739 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:13.398407    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cbb295-c661-43ea-bc33-8cbae904fa7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-rszts\" (UID: \"62cbb295-c661-43ea-bc33-8cbae904fa7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts"
Oct 31 00:13:13.398739 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:13.398617    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:13:13.398739 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:13.398686    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:13:13.400247 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:13.400210    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts"]
Oct 31 00:13:13.448059 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:13.448024    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:13:13.448059 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:13.448053    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:13:13.498884 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:13.498851    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8gw2\" (UniqueName: \"kubernetes.io/projected/62cbb295-c661-43ea-bc33-8cbae904fa7e-kube-api-access-h8gw2\") pod \"maas-api-revoke-sub-452b6d7b-rszts\" (UID: \"62cbb295-c661-43ea-bc33-8cbae904fa7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts"
Oct 31 00:13:13.499074 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:13.498917    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cbb295-c661-43ea-bc33-8cbae904fa7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-rszts\" (UID: \"62cbb295-c661-43ea-bc33-8cbae904fa7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts"
Oct 31 00:13:13.499074 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:13.499010    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cbb295-c661-43ea-bc33-8cbae904fa7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-rszts\" (UID: \"62cbb295-c661-43ea-bc33-8cbae904fa7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts"
Oct 31 00:13:13.499748 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:13.499657    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cbb295-c661-43ea-bc33-8cbae904fa7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-rszts\" (UID: \"62cbb295-c661-43ea-bc33-8cbae904fa7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts"
Oct 31 00:13:13.501297 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:13.501277    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cbb295-c661-43ea-bc33-8cbae904fa7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-rszts\" (UID: \"62cbb295-c661-43ea-bc33-8cbae904fa7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts"
Oct 31 00:13:13.555665 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:13.555634    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8gw2\" (UniqueName: \"kubernetes.io/projected/62cbb295-c661-43ea-bc33-8cbae904fa7e-kube-api-access-h8gw2\") pod \"maas-api-revoke-sub-452b6d7b-rszts\" (UID: \"62cbb295-c661-43ea-bc33-8cbae904fa7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts"
Oct 31 00:13:13.657449 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:13.657366    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts"
Oct 31 00:13:14.027214 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:14.027189    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts"]
Oct 31 00:13:14.029805 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:13:14.029772    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62cbb295_c661_43ea_bc33_8cbae904fa7e.slice/crio-d32663054073afacb01f9113e1bcd9c2db9c575a14d9341a526c9ee949a416a4 WatchSource:0}: Error finding container d32663054073afacb01f9113e1bcd9c2db9c575a14d9341a526c9ee949a416a4: Status 404 returned error can't find the container with id d32663054073afacb01f9113e1bcd9c2db9c575a14d9341a526c9ee949a416a4
Oct 31 00:13:14.570086 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:14.570050    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts" event={"ID":"62cbb295-c661-43ea-bc33-8cbae904fa7e","Type":"ContainerStarted","Data":"320d4a5021751125ecddb1e96fe70862d233c06b4fd55f8e690841f010513b4e"}
Oct 31 00:13:14.570506 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:14.570111    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts" event={"ID":"62cbb295-c661-43ea-bc33-8cbae904fa7e","Type":"ContainerStarted","Data":"d32663054073afacb01f9113e1bcd9c2db9c575a14d9341a526c9ee949a416a4"}
Oct 31 00:13:14.683782 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:14.683664    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts" podStartSLOduration=1.683650964 podStartE2EDuration="1.683650964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:14.68212664 +0000 UTC m=+3980.570032993" watchObservedRunningTime="2026-10-31 00:13:14.683650964 +0000 UTC m=+3980.571557316"
Oct 31 00:13:15.579520 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:15.579481    5078 generic.go:356] "Generic (PLEG): container finished" podID="62cbb295-c661-43ea-bc33-8cbae904fa7e" containerID="320d4a5021751125ecddb1e96fe70862d233c06b4fd55f8e690841f010513b4e" exitCode=0
Oct 31 00:13:15.580096 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:15.579539    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts" event={"ID":"62cbb295-c661-43ea-bc33-8cbae904fa7e","Type":"ContainerDied","Data":"320d4a5021751125ecddb1e96fe70862d233c06b4fd55f8e690841f010513b4e"}
Oct 31 00:13:17.341624 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:17.336375    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts"
Oct 31 00:13:17.541191 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:17.541153    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62cbb295-c661-43ea-bc33-8cbae904fa7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cbb295-c661-43ea-bc33-8cbae904fa7e-maas-api-xks-ca\") pod \"62cbb295-c661-43ea-bc33-8cbae904fa7e\" (UID: \"62cbb295-c661-43ea-bc33-8cbae904fa7e\") "
Oct 31 00:13:17.541377 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:17.541340    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62cbb295-c661-43ea-bc33-8cbae904fa7e-kube-api-access-h8gw2\" (UniqueName: \"kubernetes.io/projected/62cbb295-c661-43ea-bc33-8cbae904fa7e-kube-api-access-h8gw2\") pod \"62cbb295-c661-43ea-bc33-8cbae904fa7e\" (UID: \"62cbb295-c661-43ea-bc33-8cbae904fa7e\") "
Oct 31 00:13:17.541467 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:17.541388    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62cbb295-c661-43ea-bc33-8cbae904fa7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cbb295-c661-43ea-bc33-8cbae904fa7e-maas-api-openshift-ca\") pod \"62cbb295-c661-43ea-bc33-8cbae904fa7e\" (UID: \"62cbb295-c661-43ea-bc33-8cbae904fa7e\") "
Oct 31 00:13:17.581682 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:17.580830    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62cbb295-c661-43ea-bc33-8cbae904fa7e-maas-api-openshift-ca" pod "62cbb295-c661-43ea-bc33-8cbae904fa7e" (UID: "62cbb295-c661-43ea-bc33-8cbae904fa7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:13:17.604025 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:17.587850    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62cbb295-c661-43ea-bc33-8cbae904fa7e-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:13:17.617279 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:17.617226    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62cbb295-c661-43ea-bc33-8cbae904fa7e-maas-api-xks-ca" pod "62cbb295-c661-43ea-bc33-8cbae904fa7e" (UID: "62cbb295-c661-43ea-bc33-8cbae904fa7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:13:17.699213 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:17.696996    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62cbb295-c661-43ea-bc33-8cbae904fa7e-kube-api-access-h8gw2" pod "62cbb295-c661-43ea-bc33-8cbae904fa7e" (UID: "62cbb295-c661-43ea-bc33-8cbae904fa7e"). InnerVolumeSpecName "kube-api-access-h8gw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:13:17.708649 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:17.700884    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62cbb295-c661-43ea-bc33-8cbae904fa7e-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:13:17.708649 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:17.700917    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8gw2\" (UniqueName: \"kubernetes.io/projected/62cbb295-c661-43ea-bc33-8cbae904fa7e-kube-api-access-h8gw2\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:13:17.738663 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:17.717478    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts" event={"ID":"62cbb295-c661-43ea-bc33-8cbae904fa7e","Type":"ContainerDied","Data":"d32663054073afacb01f9113e1bcd9c2db9c575a14d9341a526c9ee949a416a4"}
Oct 31 00:13:17.738663 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:17.717540    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d32663054073afacb01f9113e1bcd9c2db9c575a14d9341a526c9ee949a416a4"
Oct 31 00:13:17.738663 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:17.717642    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts"
Oct 31 00:13:18.006434 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:13:18.006329    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:13:26.187494 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:26.187453    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"]
Oct 31 00:13:32.937242 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:32.937202    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"]
Oct 31 00:13:37.128015 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.127974    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq"]
Oct 31 00:13:37.128508 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.128315    5078 cpu_manager.go:414] "containerMap: removing container" podUID="62cbb295-c661-43ea-bc33-8cbae904fa7e" containerName="revoke-keys"
Oct 31 00:13:37.128508 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.128327    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62cbb295-c661-43ea-bc33-8cbae904fa7e" containerName="revoke-keys"
Oct 31 00:13:37.128508 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.128416    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="62cbb295-c661-43ea-bc33-8cbae904fa7e" containerName="revoke-keys"
Oct 31 00:13:37.131506 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.131483    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq"
Oct 31 00:13:37.234899 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.234867    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:13:37.235078 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.234962    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:13:37.235078 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.234971    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:13:37.235078 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.235023    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:13:37.236579 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.236557    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq"]
Oct 31 00:13:37.269147 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.269115    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af26640a-3be4-4f5a-bb6a-897b5cea489a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-6pvbq\" (UID: \"af26640a-3be4-4f5a-bb6a-897b5cea489a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq"
Oct 31 00:13:37.269287 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.269263    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltprq\" (UniqueName: \"kubernetes.io/projected/af26640a-3be4-4f5a-bb6a-897b5cea489a-kube-api-access-ltprq\") pod \"maas-api-revoke-sub-3e31a506-6pvbq\" (UID: \"af26640a-3be4-4f5a-bb6a-897b5cea489a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq"
Oct 31 00:13:37.269338 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.269292    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af26640a-3be4-4f5a-bb6a-897b5cea489a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-6pvbq\" (UID: \"af26640a-3be4-4f5a-bb6a-897b5cea489a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq"
Oct 31 00:13:37.370512 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.370475    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af26640a-3be4-4f5a-bb6a-897b5cea489a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-6pvbq\" (UID: \"af26640a-3be4-4f5a-bb6a-897b5cea489a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq"
Oct 31 00:13:37.370753 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.370554    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltprq\" (UniqueName: \"kubernetes.io/projected/af26640a-3be4-4f5a-bb6a-897b5cea489a-kube-api-access-ltprq\") pod \"maas-api-revoke-sub-3e31a506-6pvbq\" (UID: \"af26640a-3be4-4f5a-bb6a-897b5cea489a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq"
Oct 31 00:13:37.370753 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.370653    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af26640a-3be4-4f5a-bb6a-897b5cea489a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-6pvbq\" (UID: \"af26640a-3be4-4f5a-bb6a-897b5cea489a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq"
Oct 31 00:13:37.371153 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.371134    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af26640a-3be4-4f5a-bb6a-897b5cea489a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-6pvbq\" (UID: \"af26640a-3be4-4f5a-bb6a-897b5cea489a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq"
Oct 31 00:13:37.373143 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.373115    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af26640a-3be4-4f5a-bb6a-897b5cea489a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-6pvbq\" (UID: \"af26640a-3be4-4f5a-bb6a-897b5cea489a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq"
Oct 31 00:13:37.428110 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.428034    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltprq\" (UniqueName: \"kubernetes.io/projected/af26640a-3be4-4f5a-bb6a-897b5cea489a-kube-api-access-ltprq\") pod \"maas-api-revoke-sub-3e31a506-6pvbq\" (UID: \"af26640a-3be4-4f5a-bb6a-897b5cea489a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq"
Oct 31 00:13:37.443907 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.443880    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq"
Oct 31 00:13:37.819384 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.819357    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq"]
Oct 31 00:13:37.821917 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:13:37.821876    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf26640a_3be4_4f5a_bb6a_897b5cea489a.slice/crio-59664c85291b57675d36c2030c78479d347d81b353c5ce1b115d2d4c25766ec5 WatchSource:0}: Error finding container 59664c85291b57675d36c2030c78479d347d81b353c5ce1b115d2d4c25766ec5: Status 404 returned error can't find the container with id 59664c85291b57675d36c2030c78479d347d81b353c5ce1b115d2d4c25766ec5
Oct 31 00:13:37.867675 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:37.867644    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq" event={"ID":"af26640a-3be4-4f5a-bb6a-897b5cea489a","Type":"ContainerStarted","Data":"59664c85291b57675d36c2030c78479d347d81b353c5ce1b115d2d4c25766ec5"}
Oct 31 00:13:38.874967 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:38.874882    5078 generic.go:356] "Generic (PLEG): container finished" podID="af26640a-3be4-4f5a-bb6a-897b5cea489a" containerID="133037770bcf6c89780bd3c0c36d6dc290b37d6db9e5ba7f17a6369a1aa56cc4" exitCode=0
Oct 31 00:13:38.875364 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:38.874969    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq" event={"ID":"af26640a-3be4-4f5a-bb6a-897b5cea489a","Type":"ContainerDied","Data":"133037770bcf6c89780bd3c0c36d6dc290b37d6db9e5ba7f17a6369a1aa56cc4"}
Oct 31 00:13:40.519101 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:40.518771    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq"
Oct 31 00:13:40.658109 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:40.658073    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af26640a-3be4-4f5a-bb6a-897b5cea489a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af26640a-3be4-4f5a-bb6a-897b5cea489a-maas-api-openshift-ca\") pod \"af26640a-3be4-4f5a-bb6a-897b5cea489a\" (UID: \"af26640a-3be4-4f5a-bb6a-897b5cea489a\") "
Oct 31 00:13:40.659632 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:40.659590    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af26640a-3be4-4f5a-bb6a-897b5cea489a-kube-api-access-ltprq\" (UniqueName: \"kubernetes.io/projected/af26640a-3be4-4f5a-bb6a-897b5cea489a-kube-api-access-ltprq\") pod \"af26640a-3be4-4f5a-bb6a-897b5cea489a\" (UID: \"af26640a-3be4-4f5a-bb6a-897b5cea489a\") "
Oct 31 00:13:40.660238 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:40.660219    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af26640a-3be4-4f5a-bb6a-897b5cea489a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af26640a-3be4-4f5a-bb6a-897b5cea489a-maas-api-xks-ca\") pod \"af26640a-3be4-4f5a-bb6a-897b5cea489a\" (UID: \"af26640a-3be4-4f5a-bb6a-897b5cea489a\") "
Oct 31 00:13:40.660995 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:40.659462    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af26640a-3be4-4f5a-bb6a-897b5cea489a-maas-api-openshift-ca" pod "af26640a-3be4-4f5a-bb6a-897b5cea489a" (UID: "af26640a-3be4-4f5a-bb6a-897b5cea489a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:13:40.707683 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:40.706742    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af26640a-3be4-4f5a-bb6a-897b5cea489a-maas-api-xks-ca" pod "af26640a-3be4-4f5a-bb6a-897b5cea489a" (UID: "af26640a-3be4-4f5a-bb6a-897b5cea489a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:13:40.708029 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:40.707990    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af26640a-3be4-4f5a-bb6a-897b5cea489a-kube-api-access-ltprq" pod "af26640a-3be4-4f5a-bb6a-897b5cea489a" (UID: "af26640a-3be4-4f5a-bb6a-897b5cea489a"). InnerVolumeSpecName "kube-api-access-ltprq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:13:40.773623 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:40.761643    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af26640a-3be4-4f5a-bb6a-897b5cea489a-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:13:40.773623 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:40.761677    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af26640a-3be4-4f5a-bb6a-897b5cea489a-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:13:40.773623 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:40.761695    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltprq\" (UniqueName: \"kubernetes.io/projected/af26640a-3be4-4f5a-bb6a-897b5cea489a-kube-api-access-ltprq\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:13:40.922510 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:40.922468    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq" event={"ID":"af26640a-3be4-4f5a-bb6a-897b5cea489a","Type":"ContainerDied","Data":"59664c85291b57675d36c2030c78479d347d81b353c5ce1b115d2d4c25766ec5"}
Oct 31 00:13:40.925814 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:40.925788    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59664c85291b57675d36c2030c78479d347d81b353c5ce1b115d2d4c25766ec5"
Oct 31 00:13:40.926018 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:40.922866    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq"
Oct 31 00:13:41.212010 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.211967    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6" podUID="6b2368ac-55f6-496b-978c-3feed0d833fb" containerName="main" containerID="cri-o://af9b845ac5bf1697d6073ca5a5a209c0321a77ff9c736ed0716e728f15f50680" gracePeriod=45
Oct 31 00:13:41.276787 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.276404    5078 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6" podUID="6b2368ac-55f6-496b-978c-3feed0d833fb" containerName="main" probeResult="failure" output="Get \"https://10.133.1.33:8000/ready\": dial tcp 10.133.1.33:8000: connect: connection refused"
Oct 31 00:13:41.760030 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.760006    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:13:41.796767 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.796729    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-dshm\") pod \"6b2368ac-55f6-496b-978c-3feed0d833fb\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") "
Oct 31 00:13:41.796937 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.796785    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b2368ac-55f6-496b-978c-3feed0d833fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2368ac-55f6-496b-978c-3feed0d833fb-tls-certs\") pod \"6b2368ac-55f6-496b-978c-3feed0d833fb\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") "
Oct 31 00:13:41.796937 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.796881    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-home\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-home\") pod \"6b2368ac-55f6-496b-978c-3feed0d833fb\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") "
Oct 31 00:13:41.796937 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.796924    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-model-cache\") pod \"6b2368ac-55f6-496b-978c-3feed0d833fb\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") "
Oct 31 00:13:41.797125 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.796965    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-tmp-dir\") pod \"6b2368ac-55f6-496b-978c-3feed0d833fb\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") "
Oct 31 00:13:41.797125 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.797001    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b2368ac-55f6-496b-978c-3feed0d833fb-kube-api-access-zmmvq\" (UniqueName: \"kubernetes.io/projected/6b2368ac-55f6-496b-978c-3feed0d833fb-kube-api-access-zmmvq\") pod \"6b2368ac-55f6-496b-978c-3feed0d833fb\" (UID: \"6b2368ac-55f6-496b-978c-3feed0d833fb\") "
Oct 31 00:13:41.797254 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.797182    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-home" pod "6b2368ac-55f6-496b-978c-3feed0d833fb" (UID: "6b2368ac-55f6-496b-978c-3feed0d833fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:13:41.797254 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.797200    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-model-cache" pod "6b2368ac-55f6-496b-978c-3feed0d833fb" (UID: "6b2368ac-55f6-496b-978c-3feed0d833fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:13:41.797390 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.797368    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:13:41.797448 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.797397    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:13:41.797448 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.797395    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-tmp-dir" pod "6b2368ac-55f6-496b-978c-3feed0d833fb" (UID: "6b2368ac-55f6-496b-978c-3feed0d833fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:13:41.799522 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.799496    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b2368ac-55f6-496b-978c-3feed0d833fb-kube-api-access-zmmvq" pod "6b2368ac-55f6-496b-978c-3feed0d833fb" (UID: "6b2368ac-55f6-496b-978c-3feed0d833fb"). InnerVolumeSpecName "kube-api-access-zmmvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:13:41.799684 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.799624    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-dshm" pod "6b2368ac-55f6-496b-978c-3feed0d833fb" (UID: "6b2368ac-55f6-496b-978c-3feed0d833fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:13:41.799684 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.799625    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b2368ac-55f6-496b-978c-3feed0d833fb-tls-certs" pod "6b2368ac-55f6-496b-978c-3feed0d833fb" (UID: "6b2368ac-55f6-496b-978c-3feed0d833fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:13:41.897842 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.897810    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:13:41.897842 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.897842    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmmvq\" (UniqueName: \"kubernetes.io/projected/6b2368ac-55f6-496b-978c-3feed0d833fb-kube-api-access-zmmvq\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:13:41.898054 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.897853    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b2368ac-55f6-496b-978c-3feed0d833fb-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:13:41.898054 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.897862    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b2368ac-55f6-496b-978c-3feed0d833fb-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:13:41.928551 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.928518    5078 generic.go:356] "Generic (PLEG): container finished" podID="6b2368ac-55f6-496b-978c-3feed0d833fb" containerID="af9b845ac5bf1697d6073ca5a5a209c0321a77ff9c736ed0716e728f15f50680" exitCode=0
Oct 31 00:13:41.928740 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.928580    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6" event={"ID":"6b2368ac-55f6-496b-978c-3feed0d833fb","Type":"ContainerDied","Data":"af9b845ac5bf1697d6073ca5a5a209c0321a77ff9c736ed0716e728f15f50680"}
Oct 31 00:13:41.928740 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.928584    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"
Oct 31 00:13:41.928740 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.928643    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6" event={"ID":"6b2368ac-55f6-496b-978c-3feed0d833fb","Type":"ContainerDied","Data":"02837588bd955529f59e6ffd1665c7808699f6782f30016496491601289e1c9c"}
Oct 31 00:13:41.928740 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.928661    5078 scope.go:122] "RemoveContainer" containerID="af9b845ac5bf1697d6073ca5a5a209c0321a77ff9c736ed0716e728f15f50680"
Oct 31 00:13:41.938857 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.938839    5078 scope.go:122] "RemoveContainer" containerID="af9b845ac5bf1697d6073ca5a5a209c0321a77ff9c736ed0716e728f15f50680"
Oct 31 00:13:41.939165 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:13:41.939146    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af9b845ac5bf1697d6073ca5a5a209c0321a77ff9c736ed0716e728f15f50680\": container with ID starting with af9b845ac5bf1697d6073ca5a5a209c0321a77ff9c736ed0716e728f15f50680 not found: ID does not exist" containerID="af9b845ac5bf1697d6073ca5a5a209c0321a77ff9c736ed0716e728f15f50680"
Oct 31 00:13:41.939216 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:41.939175    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af9b845ac5bf1697d6073ca5a5a209c0321a77ff9c736ed0716e728f15f50680"} err="failed to get container status \"af9b845ac5bf1697d6073ca5a5a209c0321a77ff9c736ed0716e728f15f50680\": rpc error: code = NotFound desc = could not find container \"af9b845ac5bf1697d6073ca5a5a209c0321a77ff9c736ed0716e728f15f50680\": container with ID starting with af9b845ac5bf1697d6073ca5a5a209c0321a77ff9c736ed0716e728f15f50680 not found: ID does not exist"
Oct 31 00:13:42.091607 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:42.091561    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"]
Oct 31 00:13:42.093238 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:42.093213    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-ff863c73-kserve-69566c959c-z97r6"]
Oct 31 00:13:42.296060 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:42.296032    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b2368ac-55f6-496b-978c-3feed0d833fb" path="/var/lib/kubelet/pods/6b2368ac-55f6-496b-978c-3feed0d833fb/volumes"
Oct 31 00:13:47.589882 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:13:47.589849    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:13:48.057988 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.057953    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9" podUID="d4045f91-56ef-4295-a487-a4ffb741d831" containerName="main" containerID="cri-o://fabe2e63b72bb26901dfa1c0dc69ce18cdc412193066d7a477cbb55697bc1ac9" gracePeriod=45
Oct 31 00:13:48.309056 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.308999    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:13:48.355858 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.355824    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-home\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-home\") pod \"d4045f91-56ef-4295-a487-a4ffb741d831\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") "
Oct 31 00:13:48.356013 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.355888    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-model-cache\") pod \"d4045f91-56ef-4295-a487-a4ffb741d831\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") "
Oct 31 00:13:48.356013 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.355914    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-tmp-dir\") pod \"d4045f91-56ef-4295-a487-a4ffb741d831\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") "
Oct 31 00:13:48.356013 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.355932    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4045f91-56ef-4295-a487-a4ffb741d831-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4045f91-56ef-4295-a487-a4ffb741d831-tls-certs\") pod \"d4045f91-56ef-4295-a487-a4ffb741d831\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") "
Oct 31 00:13:48.356013 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.355999    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4045f91-56ef-4295-a487-a4ffb741d831-kube-api-access-kjctp\" (UniqueName: \"kubernetes.io/projected/d4045f91-56ef-4295-a487-a4ffb741d831-kube-api-access-kjctp\") pod \"d4045f91-56ef-4295-a487-a4ffb741d831\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") "
Oct 31 00:13:48.356196 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.356050    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-dshm\") pod \"d4045f91-56ef-4295-a487-a4ffb741d831\" (UID: \"d4045f91-56ef-4295-a487-a4ffb741d831\") "
Oct 31 00:13:48.356196 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.356109    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-home" pod "d4045f91-56ef-4295-a487-a4ffb741d831" (UID: "d4045f91-56ef-4295-a487-a4ffb741d831"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:13:48.356313 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.356190    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-tmp-dir" pod "d4045f91-56ef-4295-a487-a4ffb741d831" (UID: "d4045f91-56ef-4295-a487-a4ffb741d831"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:13:48.356313 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.356209    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-model-cache" pod "d4045f91-56ef-4295-a487-a4ffb741d831" (UID: "d4045f91-56ef-4295-a487-a4ffb741d831"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:13:48.356503 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.356485    5078 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-home\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:13:48.356564 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.356509    5078 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-model-cache\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:13:48.356564 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.356523    5078 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-tmp-dir\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:13:48.358288 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.358257    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-dshm" pod "d4045f91-56ef-4295-a487-a4ffb741d831" (UID: "d4045f91-56ef-4295-a487-a4ffb741d831"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:13:48.358388 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.358296    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4045f91-56ef-4295-a487-a4ffb741d831-tls-certs" pod "d4045f91-56ef-4295-a487-a4ffb741d831" (UID: "d4045f91-56ef-4295-a487-a4ffb741d831"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:13:48.358437 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.358384    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4045f91-56ef-4295-a487-a4ffb741d831-kube-api-access-kjctp" pod "d4045f91-56ef-4295-a487-a4ffb741d831" (UID: "d4045f91-56ef-4295-a487-a4ffb741d831"). InnerVolumeSpecName "kube-api-access-kjctp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:13:48.457170 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.457135    5078 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4045f91-56ef-4295-a487-a4ffb741d831-tls-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:13:48.457170 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.457167    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjctp\" (UniqueName: \"kubernetes.io/projected/d4045f91-56ef-4295-a487-a4ffb741d831-kube-api-access-kjctp\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:13:48.457170 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.457177    5078 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4045f91-56ef-4295-a487-a4ffb741d831-dshm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:13:48.964696 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.964549    5078 generic.go:356] "Generic (PLEG): container finished" podID="d4045f91-56ef-4295-a487-a4ffb741d831" containerID="fabe2e63b72bb26901dfa1c0dc69ce18cdc412193066d7a477cbb55697bc1ac9" exitCode=0
Oct 31 00:13:48.965534 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.965257    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9" event={"ID":"d4045f91-56ef-4295-a487-a4ffb741d831","Type":"ContainerDied","Data":"fabe2e63b72bb26901dfa1c0dc69ce18cdc412193066d7a477cbb55697bc1ac9"}
Oct 31 00:13:48.965534 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.965309    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9" event={"ID":"d4045f91-56ef-4295-a487-a4ffb741d831","Type":"ContainerDied","Data":"15b9872ea974b8b246a0aa6b9c7d95917ec53c68eddccba0252ad3052ef65627"}
Oct 31 00:13:48.965534 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.965343    5078 scope.go:122] "RemoveContainer" containerID="fabe2e63b72bb26901dfa1c0dc69ce18cdc412193066d7a477cbb55697bc1ac9"
Oct 31 00:13:48.965982 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.965935    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"
Oct 31 00:13:48.974951 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.974930    5078 scope.go:122] "RemoveContainer" containerID="fabe2e63b72bb26901dfa1c0dc69ce18cdc412193066d7a477cbb55697bc1ac9"
Oct 31 00:13:48.975253 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:13:48.975232    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fabe2e63b72bb26901dfa1c0dc69ce18cdc412193066d7a477cbb55697bc1ac9\": container with ID starting with fabe2e63b72bb26901dfa1c0dc69ce18cdc412193066d7a477cbb55697bc1ac9 not found: ID does not exist" containerID="fabe2e63b72bb26901dfa1c0dc69ce18cdc412193066d7a477cbb55697bc1ac9"
Oct 31 00:13:48.975352 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:48.975261    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fabe2e63b72bb26901dfa1c0dc69ce18cdc412193066d7a477cbb55697bc1ac9"} err="failed to get container status \"fabe2e63b72bb26901dfa1c0dc69ce18cdc412193066d7a477cbb55697bc1ac9\": rpc error: code = NotFound desc = could not find container \"fabe2e63b72bb26901dfa1c0dc69ce18cdc412193066d7a477cbb55697bc1ac9\": container with ID starting with fabe2e63b72bb26901dfa1c0dc69ce18cdc412193066d7a477cbb55697bc1ac9 not found: ID does not exist"
Oct 31 00:13:49.130335 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:49.130291    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"]
Oct 31 00:13:49.179792 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:49.179757    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-ff863c73-kserve-77d5cff945-dkdw9"]
Oct 31 00:13:50.296711 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:13:50.296680    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4045f91-56ef-4295-a487-a4ffb741d831" path="/var/lib/kubelet/pods/d4045f91-56ef-4295-a487-a4ffb741d831/volumes"
Oct 31 00:14:04.529713 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:04.529683    5078 scope.go:122] "RemoveContainer" containerID="cd317bb3aabdc757cd7700bde1ca04e573297113860cf5da9d28bdecb15508c1"
Oct 31 00:14:06.723179 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.723141    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4"]
Oct 31 00:14:06.723741 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.723722    5078 cpu_manager.go:414] "containerMap: removing container" podUID="af26640a-3be4-4f5a-bb6a-897b5cea489a" containerName="revoke-keys"
Oct 31 00:14:06.723809 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.723746    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af26640a-3be4-4f5a-bb6a-897b5cea489a" containerName="revoke-keys"
Oct 31 00:14:06.723809 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.723762    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d4045f91-56ef-4295-a487-a4ffb741d831" containerName="main"
Oct 31 00:14:06.723809 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.723767    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4045f91-56ef-4295-a487-a4ffb741d831" containerName="main"
Oct 31 00:14:06.723809 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.723784    5078 cpu_manager.go:414] "containerMap: removing container" podUID="6b2368ac-55f6-496b-978c-3feed0d833fb" containerName="main"
Oct 31 00:14:06.723809 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.723791    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b2368ac-55f6-496b-978c-3feed0d833fb" containerName="main"
Oct 31 00:14:06.723984 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.723881    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="af26640a-3be4-4f5a-bb6a-897b5cea489a" containerName="revoke-keys"
Oct 31 00:14:06.723984 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.723891    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4045f91-56ef-4295-a487-a4ffb741d831" containerName="main"
Oct 31 00:14:06.723984 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.723899    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b2368ac-55f6-496b-978c-3feed0d833fb" containerName="main"
Oct 31 00:14:06.728084 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.728064    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4"
Oct 31 00:14:06.780966 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.780929    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:14:06.829504 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.829470    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f13cdf0-9e63-4588-99cb-76f47ed0555c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60ed7e88-jtxf4\" (UID: \"4f13cdf0-9e63-4588-99cb-76f47ed0555c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4"
Oct 31 00:14:06.829504 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.829509    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhvs9\" (UniqueName: \"kubernetes.io/projected/4f13cdf0-9e63-4588-99cb-76f47ed0555c-kube-api-access-mhvs9\") pod \"maas-api-revoke-sub-60ed7e88-jtxf4\" (UID: \"4f13cdf0-9e63-4588-99cb-76f47ed0555c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4"
Oct 31 00:14:06.829760 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.829548    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f13cdf0-9e63-4588-99cb-76f47ed0555c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60ed7e88-jtxf4\" (UID: \"4f13cdf0-9e63-4588-99cb-76f47ed0555c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4"
Oct 31 00:14:06.830293 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.830274    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:14:06.830462 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.830448    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-5zfhp\""
Oct 31 00:14:06.830528 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.830473    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:14:06.832216 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.832199    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4"]
Oct 31 00:14:06.930621 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.930528    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f13cdf0-9e63-4588-99cb-76f47ed0555c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60ed7e88-jtxf4\" (UID: \"4f13cdf0-9e63-4588-99cb-76f47ed0555c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4"
Oct 31 00:14:06.930823 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.930672    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f13cdf0-9e63-4588-99cb-76f47ed0555c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60ed7e88-jtxf4\" (UID: \"4f13cdf0-9e63-4588-99cb-76f47ed0555c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4"
Oct 31 00:14:06.930823 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.930705    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhvs9\" (UniqueName: \"kubernetes.io/projected/4f13cdf0-9e63-4588-99cb-76f47ed0555c-kube-api-access-mhvs9\") pod \"maas-api-revoke-sub-60ed7e88-jtxf4\" (UID: \"4f13cdf0-9e63-4588-99cb-76f47ed0555c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4"
Oct 31 00:14:06.931340 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.931320    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f13cdf0-9e63-4588-99cb-76f47ed0555c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60ed7e88-jtxf4\" (UID: \"4f13cdf0-9e63-4588-99cb-76f47ed0555c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4"
Oct 31 00:14:06.933111 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.933087    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f13cdf0-9e63-4588-99cb-76f47ed0555c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60ed7e88-jtxf4\" (UID: \"4f13cdf0-9e63-4588-99cb-76f47ed0555c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4"
Oct 31 00:14:06.988335 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:06.988260    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhvs9\" (UniqueName: \"kubernetes.io/projected/4f13cdf0-9e63-4588-99cb-76f47ed0555c-kube-api-access-mhvs9\") pod \"maas-api-revoke-sub-60ed7e88-jtxf4\" (UID: \"4f13cdf0-9e63-4588-99cb-76f47ed0555c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4"
Oct 31 00:14:07.041243 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:07.041211    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4"
Oct 31 00:14:07.409458 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:07.409424    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4"]
Oct 31 00:14:07.412371 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:14:07.412340    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f13cdf0_9e63_4588_99cb_76f47ed0555c.slice/crio-52600af9a5dfc55282494359d3666daa7d3d6a0c2f60d933aa9531f373af9b70 WatchSource:0}: Error finding container 52600af9a5dfc55282494359d3666daa7d3d6a0c2f60d933aa9531f373af9b70: Status 404 returned error can't find the container with id 52600af9a5dfc55282494359d3666daa7d3d6a0c2f60d933aa9531f373af9b70
Oct 31 00:14:08.093546 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:08.093508    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4" event={"ID":"4f13cdf0-9e63-4588-99cb-76f47ed0555c","Type":"ContainerStarted","Data":"8b2ac9cfdf2a7410d3db13ff09e2c17edff192eebaa275a6de0bb3452df4f60c"}
Oct 31 00:14:08.093940 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:08.093568    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4" event={"ID":"4f13cdf0-9e63-4588-99cb-76f47ed0555c","Type":"ContainerStarted","Data":"52600af9a5dfc55282494359d3666daa7d3d6a0c2f60d933aa9531f373af9b70"}
Oct 31 00:14:08.207239 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:08.207180    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4" podStartSLOduration=2.207147304 podStartE2EDuration="2.207147304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:08.206703838 +0000 UTC m=+4034.094610192" watchObservedRunningTime="2026-10-31 00:14:08.207147304 +0000 UTC m=+4034.095053659"
Oct 31 00:14:09.099313 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:09.099284    5078 generic.go:356] "Generic (PLEG): container finished" podID="4f13cdf0-9e63-4588-99cb-76f47ed0555c" containerID="8b2ac9cfdf2a7410d3db13ff09e2c17edff192eebaa275a6de0bb3452df4f60c" exitCode=0
Oct 31 00:14:09.099846 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:09.099333    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4" event={"ID":"4f13cdf0-9e63-4588-99cb-76f47ed0555c","Type":"ContainerDied","Data":"8b2ac9cfdf2a7410d3db13ff09e2c17edff192eebaa275a6de0bb3452df4f60c"}
Oct 31 00:14:10.236591 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:10.236564    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4"
Oct 31 00:14:10.366192 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:10.366112    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f13cdf0-9e63-4588-99cb-76f47ed0555c-kube-api-access-mhvs9\" (UniqueName: \"kubernetes.io/projected/4f13cdf0-9e63-4588-99cb-76f47ed0555c-kube-api-access-mhvs9\") pod \"4f13cdf0-9e63-4588-99cb-76f47ed0555c\" (UID: \"4f13cdf0-9e63-4588-99cb-76f47ed0555c\") "
Oct 31 00:14:10.366192 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:10.366167    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f13cdf0-9e63-4588-99cb-76f47ed0555c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f13cdf0-9e63-4588-99cb-76f47ed0555c-maas-api-openshift-ca\") pod \"4f13cdf0-9e63-4588-99cb-76f47ed0555c\" (UID: \"4f13cdf0-9e63-4588-99cb-76f47ed0555c\") "
Oct 31 00:14:10.366456 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:10.366195    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f13cdf0-9e63-4588-99cb-76f47ed0555c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f13cdf0-9e63-4588-99cb-76f47ed0555c-maas-api-xks-ca\") pod \"4f13cdf0-9e63-4588-99cb-76f47ed0555c\" (UID: \"4f13cdf0-9e63-4588-99cb-76f47ed0555c\") "
Oct 31 00:14:10.366588 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:10.366561    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f13cdf0-9e63-4588-99cb-76f47ed0555c-maas-api-openshift-ca" pod "4f13cdf0-9e63-4588-99cb-76f47ed0555c" (UID: "4f13cdf0-9e63-4588-99cb-76f47ed0555c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:14:10.368549 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:10.368517    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f13cdf0-9e63-4588-99cb-76f47ed0555c-maas-api-xks-ca" pod "4f13cdf0-9e63-4588-99cb-76f47ed0555c" (UID: "4f13cdf0-9e63-4588-99cb-76f47ed0555c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:14:10.368549 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:10.368532    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f13cdf0-9e63-4588-99cb-76f47ed0555c-kube-api-access-mhvs9" pod "4f13cdf0-9e63-4588-99cb-76f47ed0555c" (UID: "4f13cdf0-9e63-4588-99cb-76f47ed0555c"). InnerVolumeSpecName "kube-api-access-mhvs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:10.467804 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:10.467769    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhvs9\" (UniqueName: \"kubernetes.io/projected/4f13cdf0-9e63-4588-99cb-76f47ed0555c-kube-api-access-mhvs9\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:14:10.467804 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:10.467796    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f13cdf0-9e63-4588-99cb-76f47ed0555c-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:14:10.467804 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:10.467808    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f13cdf0-9e63-4588-99cb-76f47ed0555c-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:14:11.108444 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:11.108414    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4"
Oct 31 00:14:11.108687 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:11.108409    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4" event={"ID":"4f13cdf0-9e63-4588-99cb-76f47ed0555c","Type":"ContainerDied","Data":"52600af9a5dfc55282494359d3666daa7d3d6a0c2f60d933aa9531f373af9b70"}
Oct 31 00:14:11.108687 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:11.108545    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52600af9a5dfc55282494359d3666daa7d3d6a0c2f60d933aa9531f373af9b70"
Oct 31 00:14:17.044581 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:17.044540    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z"]
Oct 31 00:14:17.094122 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:17.094077    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-bdx9z"]
Oct 31 00:14:17.590132 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:14:17.590093    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:14:18.296345 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:18.296313    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2" path="/var/lib/kubelet/pods/0c1ee3ef-1403-4ee5-a61e-0bb9bfaabff2/volumes"
Oct 31 00:14:26.940138 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:26.940104    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"]
Oct 31 00:14:26.940756 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:26.940443    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4f13cdf0-9e63-4588-99cb-76f47ed0555c" containerName="revoke-keys"
Oct 31 00:14:26.940756 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:26.940456    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f13cdf0-9e63-4588-99cb-76f47ed0555c" containerName="revoke-keys"
Oct 31 00:14:26.940756 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:26.940512    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f13cdf0-9e63-4588-99cb-76f47ed0555c" containerName="revoke-keys"
Oct 31 00:14:26.943857 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:26.943839    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:26.996223 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:26.996187    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4145bafc-openshift-default-dockercfg-b4cjd\""
Oct 31 00:14:27.007940 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.007910    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-envoy\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.008068 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.007950    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56tfk\" (UniqueName: \"kubernetes.io/projected/7a53f2e0-cece-46a9-9ece-17c74293e761-kube-api-access-56tfk\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.008068 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.007981    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-data\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.008147 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.008108    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a53f2e0-cece-46a9-9ece-17c74293e761-istiod-ca-cert\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.008188 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.008158    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-workload-socket\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.008227 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.008210    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-workload-certs\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.008262 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.008245    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-credential-socket\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.008337 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.008315    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-token\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.008375 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.008356    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-podinfo\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.049326 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.049297    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"]
Oct 31 00:14:27.109181 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.109150    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-podinfo\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.109377 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.109216    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-envoy\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.109377 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.109243    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56tfk\" (UniqueName: \"kubernetes.io/projected/7a53f2e0-cece-46a9-9ece-17c74293e761-kube-api-access-56tfk\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.109377 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.109263    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-data\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.109377 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.109283    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a53f2e0-cece-46a9-9ece-17c74293e761-istiod-ca-cert\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.109377 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.109298    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-workload-socket\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.109377 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.109318    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-workload-certs\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.109377 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.109354    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-credential-socket\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.109838 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.109389    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-token\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.109838 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.109747    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-data\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.109955 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.109880    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-workload-socket\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.109955 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.109938    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-workload-certs\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.110233 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.109953    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-credential-socket\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.110233 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.110129    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a53f2e0-cece-46a9-9ece-17c74293e761-istiod-ca-cert\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.111944 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.111921    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-envoy\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.112072 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.111947    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-podinfo\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.166286 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.166258    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-token\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.166456 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.166387    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56tfk\" (UniqueName: \"kubernetes.io/projected/7a53f2e0-cece-46a9-9ece-17c74293e761-kube-api-access-56tfk\") pod \"e2e-ait-4145bafc-openshift-default-545d555477-7k44r\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") " pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.261444 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.261330    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:27.693236 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.693212    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"]
Oct 31 00:14:27.695237 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:14:27.695214    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a53f2e0_cece_46a9_9ece_17c74293e761.slice/crio-2b8d46c5e511e5c546589be792b8a5bf7fc9e935382b50ebc8807003c5685367 WatchSource:0}: Error finding container 2b8d46c5e511e5c546589be792b8a5bf7fc9e935382b50ebc8807003c5685367: Status 404 returned error can't find the container with id 2b8d46c5e511e5c546589be792b8a5bf7fc9e935382b50ebc8807003c5685367
Oct 31 00:14:27.697393 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.697360    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:14:27.697518 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.697440    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:14:27.697518 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:27.697483    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:14:28.178887 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:28.178848    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r" event={"ID":"7a53f2e0-cece-46a9-9ece-17c74293e761","Type":"ContainerStarted","Data":"88963f1eae9ac4c6aa632d320deda8196be859a2f1d2ba92ffc0df0954dcd4a3"}
Oct 31 00:14:28.179304 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:28.178912    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r" event={"ID":"7a53f2e0-cece-46a9-9ece-17c74293e761","Type":"ContainerStarted","Data":"2b8d46c5e511e5c546589be792b8a5bf7fc9e935382b50ebc8807003c5685367"}
Oct 31 00:14:28.261923 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:28.261888    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:28.266668 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:28.266644    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:28.299339 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:28.299284    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r" podStartSLOduration=2.299265827 podStartE2EDuration="2.299265827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:28.297784237 +0000 UTC m=+4054.185690619" watchObservedRunningTime="2026-10-31 00:14:28.299265827 +0000 UTC m=+4054.187172181"
Oct 31 00:14:29.183082 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:29.183052    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:29.184099 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:29.184082    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:34.209240 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:34.203369    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7"]
Oct 31 00:14:34.213723 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:34.213690    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7"
Oct 31 00:14:34.265076 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:34.265042    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7"]
Oct 31 00:14:34.280884 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:34.280857    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455ae29b-fee0-404a-9eb5-1964c857cb8c-plugins-config-volume\") pod \"payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7\" (UID: \"455ae29b-fee0-404a-9eb5-1964c857cb8c\") " pod="openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7"
Oct 31 00:14:34.281056 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:34.280965    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjzlc\" (UniqueName: \"kubernetes.io/projected/455ae29b-fee0-404a-9eb5-1964c857cb8c-kube-api-access-vjzlc\") pod \"payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7\" (UID: \"455ae29b-fee0-404a-9eb5-1964c857cb8c\") " pod="openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7"
Oct 31 00:14:34.312532 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:34.312505    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4145bafc\""
Oct 31 00:14:34.312719 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:34.312586    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4145bafc-dockercfg-rrvwt\""
Oct 31 00:14:34.381827 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:34.381789    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjzlc\" (UniqueName: \"kubernetes.io/projected/455ae29b-fee0-404a-9eb5-1964c857cb8c-kube-api-access-vjzlc\") pod \"payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7\" (UID: \"455ae29b-fee0-404a-9eb5-1964c857cb8c\") " pod="openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7"
Oct 31 00:14:34.381827 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:34.381829    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455ae29b-fee0-404a-9eb5-1964c857cb8c-plugins-config-volume\") pod \"payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7\" (UID: \"455ae29b-fee0-404a-9eb5-1964c857cb8c\") " pod="openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7"
Oct 31 00:14:34.382541 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:34.382523    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455ae29b-fee0-404a-9eb5-1964c857cb8c-plugins-config-volume\") pod \"payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7\" (UID: \"455ae29b-fee0-404a-9eb5-1964c857cb8c\") " pod="openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7"
Oct 31 00:14:34.438807 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:34.438779    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjzlc\" (UniqueName: \"kubernetes.io/projected/455ae29b-fee0-404a-9eb5-1964c857cb8c-kube-api-access-vjzlc\") pod \"payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7\" (UID: \"455ae29b-fee0-404a-9eb5-1964c857cb8c\") " pod="openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7"
Oct 31 00:14:34.529787 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:34.529705    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7"
Oct 31 00:14:34.903555 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:34.903371    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7"]
Oct 31 00:14:34.906257 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:14:34.906224    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455ae29b_fee0_404a_9eb5_1964c857cb8c.slice/crio-fc148c2a728b7b67ca2dc4bea3d0bed7787a6c6d82135385e9fa1fff62b9676f WatchSource:0}: Error finding container fc148c2a728b7b67ca2dc4bea3d0bed7787a6c6d82135385e9fa1fff62b9676f: Status 404 returned error can't find the container with id fc148c2a728b7b67ca2dc4bea3d0bed7787a6c6d82135385e9fa1fff62b9676f
Oct 31 00:14:35.218834 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:35.218749    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7" event={"ID":"455ae29b-fee0-404a-9eb5-1964c857cb8c","Type":"ContainerStarted","Data":"29a22fb50ce9c30318485db18ad4f25c091134183b9fd29f22aa59b8ac896b37"}
Oct 31 00:14:35.218834 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:35.218807    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7" event={"ID":"455ae29b-fee0-404a-9eb5-1964c857cb8c","Type":"ContainerStarted","Data":"fc148c2a728b7b67ca2dc4bea3d0bed7787a6c6d82135385e9fa1fff62b9676f"}
Oct 31 00:14:35.219310 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:35.218858    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7"
Oct 31 00:14:35.332042 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:35.331991    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7" podStartSLOduration=1.331975757 podStartE2EDuration="1.331975757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:35.330734529 +0000 UTC m=+4061.218640876" watchObservedRunningTime="2026-10-31 00:14:35.331975757 +0000 UTC m=+4061.219882101"
Oct 31 00:14:46.044474 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:46.044441    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t"]
Oct 31 00:14:46.093917 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:46.093856    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-dbr6t"]
Oct 31 00:14:46.227333 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:46.227307    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7"
Oct 31 00:14:46.296474 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:46.296382    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28092b6f-7204-4d92-82b7-3b4caece1cb9" path="/var/lib/kubelet/pods/28092b6f-7204-4d92-82b7-3b4caece1cb9/volumes"
Oct 31 00:14:47.606563 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:14:47.606527    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:14:53.321892 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:53.321859    5078 generic.go:356] "Generic (PLEG): container finished" podID="7a53f2e0-cece-46a9-9ece-17c74293e761" containerID="88963f1eae9ac4c6aa632d320deda8196be859a2f1d2ba92ffc0df0954dcd4a3" exitCode=0
Oct 31 00:14:53.322421 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:53.321911    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r" event={"ID":"7a53f2e0-cece-46a9-9ece-17c74293e761","Type":"ContainerDied","Data":"88963f1eae9ac4c6aa632d320deda8196be859a2f1d2ba92ffc0df0954dcd4a3"}
Oct 31 00:14:53.322421 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:53.322254    5078 scope.go:122] "RemoveContainer" containerID="88963f1eae9ac4c6aa632d320deda8196be859a2f1d2ba92ffc0df0954dcd4a3"
Oct 31 00:14:53.323424 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:53.323396    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:14:53.323495 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:53.323480    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:14:53.323545 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:53.323519    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:14:54.327649 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:54.327591    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r" event={"ID":"7a53f2e0-cece-46a9-9ece-17c74293e761","Type":"ContainerStarted","Data":"ba70a97d60430c079b5adc6da679ee7557eb0759dde656c62fcc60439340faab"}
Oct 31 00:14:55.275623 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:55.269981    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:55.277635 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:55.277582    5078 patch_prober.go:29] interesting pod/e2e-ait-4145bafc-openshift-default-545d555477-7k44r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Oct 31 00:14:55.277906 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:55.277878    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r" podUID="7a53f2e0-cece-46a9-9ece-17c74293e761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Oct 31 00:14:56.270074 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.270033    5078 patch_prober.go:29] interesting pod/e2e-ait-4145bafc-openshift-default-545d555477-7k44r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Oct 31 00:14:56.270733 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.270695    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r" podUID="7a53f2e0-cece-46a9-9ece-17c74293e761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Oct 31 00:14:56.500426 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.500386    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx"]
Oct 31 00:14:56.503808 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.503787    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx"
Oct 31 00:14:56.616739 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.616699    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx\" (UID: \"d473b4f3-8bd6-42fc-b6aa-62570e6deb27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx"
Oct 31 00:14:56.616932 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.616749    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc8d5\" (UniqueName: \"kubernetes.io/projected/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-kube-api-access-jc8d5\") pod \"maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx\" (UID: \"d473b4f3-8bd6-42fc-b6aa-62570e6deb27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx"
Oct 31 00:14:56.616932 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.616891    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx\" (UID: \"d473b4f3-8bd6-42fc-b6aa-62570e6deb27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx"
Oct 31 00:14:56.626734 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.626706    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:14:56.626911 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.626763    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:14:56.626911 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.626815    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:14:56.626911 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.626897    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnrc9\""
Oct 31 00:14:56.628427 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.628409    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx"]
Oct 31 00:14:56.718410 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.718374    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx\" (UID: \"d473b4f3-8bd6-42fc-b6aa-62570e6deb27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx"
Oct 31 00:14:56.718410 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.718419    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc8d5\" (UniqueName: \"kubernetes.io/projected/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-kube-api-access-jc8d5\") pod \"maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx\" (UID: \"d473b4f3-8bd6-42fc-b6aa-62570e6deb27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx"
Oct 31 00:14:56.718701 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.718467    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx\" (UID: \"d473b4f3-8bd6-42fc-b6aa-62570e6deb27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx"
Oct 31 00:14:56.719079 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.719056    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx\" (UID: \"d473b4f3-8bd6-42fc-b6aa-62570e6deb27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx"
Oct 31 00:14:56.720956 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.720938    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx\" (UID: \"d473b4f3-8bd6-42fc-b6aa-62570e6deb27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx"
Oct 31 00:14:56.783805 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.783781    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc8d5\" (UniqueName: \"kubernetes.io/projected/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-kube-api-access-jc8d5\") pod \"maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx\" (UID: \"d473b4f3-8bd6-42fc-b6aa-62570e6deb27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx"
Oct 31 00:14:56.816032 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:56.815998    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx"
Oct 31 00:14:57.207439 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:57.207317    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx"]
Oct 31 00:14:57.209535 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:14:57.209502    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd473b4f3_8bd6_42fc_b6aa_62570e6deb27.slice/crio-f20f5d4ebf616583eb26e0525428e1ae5cca234d15933a8933f73e6dc4c01972 WatchSource:0}: Error finding container f20f5d4ebf616583eb26e0525428e1ae5cca234d15933a8933f73e6dc4c01972: Status 404 returned error can't find the container with id f20f5d4ebf616583eb26e0525428e1ae5cca234d15933a8933f73e6dc4c01972
Oct 31 00:14:57.276623 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:57.266248    5078 patch_prober.go:29] interesting pod/e2e-ait-4145bafc-openshift-default-545d555477-7k44r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Oct 31 00:14:57.276623 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:57.266306    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r" podUID="7a53f2e0-cece-46a9-9ece-17c74293e761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Oct 31 00:14:57.342039 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:57.342003    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx" event={"ID":"d473b4f3-8bd6-42fc-b6aa-62570e6deb27","Type":"ContainerStarted","Data":"f20f5d4ebf616583eb26e0525428e1ae5cca234d15933a8933f73e6dc4c01972"}
Oct 31 00:14:58.281774 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:58.281734    5078 patch_prober.go:29] interesting pod/e2e-ait-4145bafc-openshift-default-545d555477-7k44r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Oct 31 00:14:58.282233 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:58.281797    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r" podUID="7a53f2e0-cece-46a9-9ece-17c74293e761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Oct 31 00:14:58.348101 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:58.348058    5078 generic.go:356] "Generic (PLEG): container finished" podID="d473b4f3-8bd6-42fc-b6aa-62570e6deb27" containerID="7512928f74b7c0d979fdae94ad91c6a15281e636e5a74ca2bae0d3e7bce399c2" exitCode=0
Oct 31 00:14:58.348292 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:58.348130    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx" event={"ID":"d473b4f3-8bd6-42fc-b6aa-62570e6deb27","Type":"ContainerDied","Data":"7512928f74b7c0d979fdae94ad91c6a15281e636e5a74ca2bae0d3e7bce399c2"}
Oct 31 00:14:59.183617 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:59.183565    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:14:59.262636 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:59.262584    5078 patch_prober.go:29] interesting pod/e2e-ait-4145bafc-openshift-default-545d555477-7k44r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused" start-of-body=
Oct 31 00:14:59.262823 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:59.262655    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r" podUID="7a53f2e0-cece-46a9-9ece-17c74293e761" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.38:15021/healthz/ready\": dial tcp 10.133.1.38:15021: connect: connection refused"
Oct 31 00:14:59.494955 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:59.494922    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx"
Oct 31 00:14:59.545664 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:59.545634    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-maas-api-xks-ca\") pod \"d473b4f3-8bd6-42fc-b6aa-62570e6deb27\" (UID: \"d473b4f3-8bd6-42fc-b6aa-62570e6deb27\") "
Oct 31 00:14:59.545861 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:59.545740    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-kube-api-access-jc8d5\" (UniqueName: \"kubernetes.io/projected/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-kube-api-access-jc8d5\") pod \"d473b4f3-8bd6-42fc-b6aa-62570e6deb27\" (UID: \"d473b4f3-8bd6-42fc-b6aa-62570e6deb27\") "
Oct 31 00:14:59.545861 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:59.545774    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-maas-api-openshift-ca\") pod \"d473b4f3-8bd6-42fc-b6aa-62570e6deb27\" (UID: \"d473b4f3-8bd6-42fc-b6aa-62570e6deb27\") "
Oct 31 00:14:59.546192 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:59.546168    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-maas-api-openshift-ca" pod "d473b4f3-8bd6-42fc-b6aa-62570e6deb27" (UID: "d473b4f3-8bd6-42fc-b6aa-62570e6deb27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:14:59.548267 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:59.548238    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-maas-api-xks-ca" pod "d473b4f3-8bd6-42fc-b6aa-62570e6deb27" (UID: "d473b4f3-8bd6-42fc-b6aa-62570e6deb27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:14:59.548379 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:59.548364    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-kube-api-access-jc8d5" pod "d473b4f3-8bd6-42fc-b6aa-62570e6deb27" (UID: "d473b4f3-8bd6-42fc-b6aa-62570e6deb27"). InnerVolumeSpecName "kube-api-access-jc8d5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:59.647074 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:59.647039    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:14:59.647074 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:59.647068    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc8d5\" (UniqueName: \"kubernetes.io/projected/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-kube-api-access-jc8d5\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:14:59.647074 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:14:59.647081    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d473b4f3-8bd6-42fc-b6aa-62570e6deb27-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:15:00.155949 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.155914    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr"]
Oct 31 00:15:00.156298 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.156285    5078 cpu_manager.go:414] "containerMap: removing container" podUID="d473b4f3-8bd6-42fc-b6aa-62570e6deb27" containerName="revoke-keys"
Oct 31 00:15:00.156349 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.156299    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d473b4f3-8bd6-42fc-b6aa-62570e6deb27" containerName="revoke-keys"
Oct 31 00:15:00.156386 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.156367    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="d473b4f3-8bd6-42fc-b6aa-62570e6deb27" containerName="revoke-keys"
Oct 31 00:15:00.159385 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.159367    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr"
Oct 31 00:15:00.162335 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.162307    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45"]
Oct 31 00:15:00.165451 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.165431    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45"
Oct 31 00:15:00.215637 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.215573    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr"]
Oct 31 00:15:00.254044 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.254007    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5b19ec-9d4a-4f38-907b-3e0414b7f525-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45\" (UID: \"3c5b19ec-9d4a-4f38-907b-3e0414b7f525\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45"
Oct 31 00:15:00.254239 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.254129    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1e6bcb-4201-4905-980d-f7fd00f191c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890095-nc7hr\" (UID: \"3f1e6bcb-4201-4905-980d-f7fd00f191c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr"
Oct 31 00:15:00.254239 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.254159    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c8cm\" (UniqueName: \"kubernetes.io/projected/3c5b19ec-9d4a-4f38-907b-3e0414b7f525-kube-api-access-9c8cm\") pod \"maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45\" (UID: \"3c5b19ec-9d4a-4f38-907b-3e0414b7f525\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45"
Oct 31 00:15:00.254380 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.254306    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9n8d\" (UniqueName: \"kubernetes.io/projected/3f1e6bcb-4201-4905-980d-f7fd00f191c3-kube-api-access-z9n8d\") pod \"maas-api-key-cleanup-29890095-nc7hr\" (UID: \"3f1e6bcb-4201-4905-980d-f7fd00f191c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr"
Oct 31 00:15:00.257581 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.257552    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45"]
Oct 31 00:15:00.266946 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.266922    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:15:00.268126 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.268106    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:15:00.355029 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.354985    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1e6bcb-4201-4905-980d-f7fd00f191c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890095-nc7hr\" (UID: \"3f1e6bcb-4201-4905-980d-f7fd00f191c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr"
Oct 31 00:15:00.355029 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.355021    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c8cm\" (UniqueName: \"kubernetes.io/projected/3c5b19ec-9d4a-4f38-907b-3e0414b7f525-kube-api-access-9c8cm\") pod \"maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45\" (UID: \"3c5b19ec-9d4a-4f38-907b-3e0414b7f525\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45"
Oct 31 00:15:00.355320 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.355246    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9n8d\" (UniqueName: \"kubernetes.io/projected/3f1e6bcb-4201-4905-980d-f7fd00f191c3-kube-api-access-z9n8d\") pod \"maas-api-key-cleanup-29890095-nc7hr\" (UID: \"3f1e6bcb-4201-4905-980d-f7fd00f191c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr"
Oct 31 00:15:00.355388 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.355333    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5b19ec-9d4a-4f38-907b-3e0414b7f525-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45\" (UID: \"3c5b19ec-9d4a-4f38-907b-3e0414b7f525\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45"
Oct 31 00:15:00.355840 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.355817    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1e6bcb-4201-4905-980d-f7fd00f191c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890095-nc7hr\" (UID: \"3f1e6bcb-4201-4905-980d-f7fd00f191c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr"
Oct 31 00:15:00.356271 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.356250    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5b19ec-9d4a-4f38-907b-3e0414b7f525-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45\" (UID: \"3c5b19ec-9d4a-4f38-907b-3e0414b7f525\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45"
Oct 31 00:15:00.360023 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.360003    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx"
Oct 31 00:15:00.360124 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.360015    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx" event={"ID":"d473b4f3-8bd6-42fc-b6aa-62570e6deb27","Type":"ContainerDied","Data":"f20f5d4ebf616583eb26e0525428e1ae5cca234d15933a8933f73e6dc4c01972"}
Oct 31 00:15:00.360124 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.360066    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f20f5d4ebf616583eb26e0525428e1ae5cca234d15933a8933f73e6dc4c01972"
Oct 31 00:15:00.416008 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.415932    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9n8d\" (UniqueName: \"kubernetes.io/projected/3f1e6bcb-4201-4905-980d-f7fd00f191c3-kube-api-access-z9n8d\") pod \"maas-api-key-cleanup-29890095-nc7hr\" (UID: \"3f1e6bcb-4201-4905-980d-f7fd00f191c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr"
Oct 31 00:15:00.416295 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.416244    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c8cm\" (UniqueName: \"kubernetes.io/projected/3c5b19ec-9d4a-4f38-907b-3e0414b7f525-kube-api-access-9c8cm\") pod \"maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45\" (UID: \"3c5b19ec-9d4a-4f38-907b-3e0414b7f525\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45"
Oct 31 00:15:00.477205 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.477177    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr"
Oct 31 00:15:00.485420 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.485397    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45"
Oct 31 00:15:00.857441 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.857415    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr"]
Oct 31 00:15:00.859611 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:15:00.859572    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f1e6bcb_4201_4905_980d_f7fd00f191c3.slice/crio-9134ca137adb61ec6874cfa9619797a97a7ac9dd027dc9f135ce57ac8cb896f9 WatchSource:0}: Error finding container 9134ca137adb61ec6874cfa9619797a97a7ac9dd027dc9f135ce57ac8cb896f9: Status 404 returned error can't find the container with id 9134ca137adb61ec6874cfa9619797a97a7ac9dd027dc9f135ce57ac8cb896f9
Oct 31 00:15:00.906654 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:00.906627    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45"]
Oct 31 00:15:00.909523 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:15:00.909500    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5b19ec_9d4a_4f38_907b_3e0414b7f525.slice/crio-5db607b4d63c25d087b78009ae2da618e2b7ce233c25703ac06c2e591be4987a WatchSource:0}: Error finding container 5db607b4d63c25d087b78009ae2da618e2b7ce233c25703ac06c2e591be4987a: Status 404 returned error can't find the container with id 5db607b4d63c25d087b78009ae2da618e2b7ce233c25703ac06c2e591be4987a
Oct 31 00:15:01.366742 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:01.366708    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr" event={"ID":"3f1e6bcb-4201-4905-980d-f7fd00f191c3","Type":"ContainerStarted","Data":"3cd6cf894188a2a88cae4c970e161db96cdc6ea2b960375d0ae40b34421410a1"}
Oct 31 00:15:01.366946 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:01.366770    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr" event={"ID":"3f1e6bcb-4201-4905-980d-f7fd00f191c3","Type":"ContainerStarted","Data":"9134ca137adb61ec6874cfa9619797a97a7ac9dd027dc9f135ce57ac8cb896f9"}
Oct 31 00:15:01.368334 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:01.368312    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45" event={"ID":"3c5b19ec-9d4a-4f38-907b-3e0414b7f525","Type":"ContainerStarted","Data":"faac8d1429de30442e3383f0442869d540d29ab4ea4f9077573f888a69e6d1be"}
Oct 31 00:15:01.368458 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:01.368349    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45" event={"ID":"3c5b19ec-9d4a-4f38-907b-3e0414b7f525","Type":"ContainerStarted","Data":"5db607b4d63c25d087b78009ae2da618e2b7ce233c25703ac06c2e591be4987a"}
Oct 31 00:15:01.479553 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:01.479450    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr" podStartSLOduration=1.479434989 podStartE2EDuration="1.479434989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:01.478386042 +0000 UTC m=+4087.366292396" watchObservedRunningTime="2026-10-31 00:15:01.479434989 +0000 UTC m=+4087.367341342"
Oct 31 00:15:01.593879 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:01.593824    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45" podStartSLOduration=1.593809613 podStartE2EDuration="1.593809613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:01.592071609 +0000 UTC m=+4087.479977963" watchObservedRunningTime="2026-10-31 00:15:01.593809613 +0000 UTC m=+4087.481715966"
Oct 31 00:15:02.374718 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:02.374685    5078 generic.go:356] "Generic (PLEG): container finished" podID="3f1e6bcb-4201-4905-980d-f7fd00f191c3" containerID="3cd6cf894188a2a88cae4c970e161db96cdc6ea2b960375d0ae40b34421410a1" exitCode=60
Oct 31 00:15:02.375217 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:02.374772    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr" event={"ID":"3f1e6bcb-4201-4905-980d-f7fd00f191c3","Type":"ContainerDied","Data":"3cd6cf894188a2a88cae4c970e161db96cdc6ea2b960375d0ae40b34421410a1"}
Oct 31 00:15:02.375217 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:02.374977    5078 scope.go:122] "RemoveContainer" containerID="3cd6cf894188a2a88cae4c970e161db96cdc6ea2b960375d0ae40b34421410a1"
Oct 31 00:15:02.376426 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:02.376408    5078 generic.go:356] "Generic (PLEG): container finished" podID="3c5b19ec-9d4a-4f38-907b-3e0414b7f525" containerID="faac8d1429de30442e3383f0442869d540d29ab4ea4f9077573f888a69e6d1be" exitCode=60
Oct 31 00:15:02.376505 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:02.376466    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45" event={"ID":"3c5b19ec-9d4a-4f38-907b-3e0414b7f525","Type":"ContainerDied","Data":"faac8d1429de30442e3383f0442869d540d29ab4ea4f9077573f888a69e6d1be"}
Oct 31 00:15:02.376699 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:02.376685    5078 scope.go:122] "RemoveContainer" containerID="faac8d1429de30442e3383f0442869d540d29ab4ea4f9077573f888a69e6d1be"
Oct 31 00:15:03.385934 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:03.385893    5078 generic.go:356] "Generic (PLEG): container finished" podID="3f1e6bcb-4201-4905-980d-f7fd00f191c3" containerID="b9ae0aab745adf2c27106d3fe66d8e736a338bf65cab983a6f0dd6be52443ee3" exitCode=60
Oct 31 00:15:03.386483 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:03.385964    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr" event={"ID":"3f1e6bcb-4201-4905-980d-f7fd00f191c3","Type":"ContainerDied","Data":"b9ae0aab745adf2c27106d3fe66d8e736a338bf65cab983a6f0dd6be52443ee3"}
Oct 31 00:15:03.386483 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:03.386021    5078 scope.go:122] "RemoveContainer" containerID="3cd6cf894188a2a88cae4c970e161db96cdc6ea2b960375d0ae40b34421410a1"
Oct 31 00:15:03.386483 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:03.386236    5078 scope.go:122] "RemoveContainer" containerID="b9ae0aab745adf2c27106d3fe66d8e736a338bf65cab983a6f0dd6be52443ee3"
Oct 31 00:15:03.386483 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:15:03.386436    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890095-nc7hr_odh-ai-gateway-infra(3f1e6bcb-4201-4905-980d-f7fd00f191c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr" podUID="3f1e6bcb-4201-4905-980d-f7fd00f191c3"
Oct 31 00:15:03.388307 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:03.388281    5078 generic.go:356] "Generic (PLEG): container finished" podID="3c5b19ec-9d4a-4f38-907b-3e0414b7f525" containerID="301c5e1f74948f75509f4db77dbecdcce7a7896e7dd1b1a09036a38adb7b74ff" exitCode=60
Oct 31 00:15:03.388421 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:03.388332    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45" event={"ID":"3c5b19ec-9d4a-4f38-907b-3e0414b7f525","Type":"ContainerDied","Data":"301c5e1f74948f75509f4db77dbecdcce7a7896e7dd1b1a09036a38adb7b74ff"}
Oct 31 00:15:03.388535 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:03.388522    5078 scope.go:122] "RemoveContainer" containerID="301c5e1f74948f75509f4db77dbecdcce7a7896e7dd1b1a09036a38adb7b74ff"
Oct 31 00:15:03.388739 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:15:03.388720    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45_odh-ai-gateway-infra(3c5b19ec-9d4a-4f38-907b-3e0414b7f525)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45" podUID="3c5b19ec-9d4a-4f38-907b-3e0414b7f525"
Oct 31 00:15:03.399517 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:03.399489    5078 scope.go:122] "RemoveContainer" containerID="faac8d1429de30442e3383f0442869d540d29ab4ea4f9077573f888a69e6d1be"
Oct 31 00:15:04.395345 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:04.395316    5078 scope.go:122] "RemoveContainer" containerID="301c5e1f74948f75509f4db77dbecdcce7a7896e7dd1b1a09036a38adb7b74ff"
Oct 31 00:15:04.395805 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:15:04.395524    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45_odh-ai-gateway-infra(3c5b19ec-9d4a-4f38-907b-3e0414b7f525)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45" podUID="3c5b19ec-9d4a-4f38-907b-3e0414b7f525"
Oct 31 00:15:04.397217 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:04.397199    5078 scope.go:122] "RemoveContainer" containerID="b9ae0aab745adf2c27106d3fe66d8e736a338bf65cab983a6f0dd6be52443ee3"
Oct 31 00:15:04.397395 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:15:04.397377    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890095-nc7hr_odh-ai-gateway-infra(3f1e6bcb-4201-4905-980d-f7fd00f191c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr" podUID="3f1e6bcb-4201-4905-980d-f7fd00f191c3"
Oct 31 00:15:08.060915 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:08.060882    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7"]
Oct 31 00:15:13.061458 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:13.061396    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7" podUID="455ae29b-fee0-404a-9eb5-1964c857cb8c" containerName="payload-processing" containerID="cri-o://29a22fb50ce9c30318485db18ad4f25c091134183b9fd29f22aa59b8ac896b37" gracePeriod=25
Oct 31 00:15:13.311201 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:13.311176    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7"
Oct 31 00:15:13.382651 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:13.382546    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/455ae29b-fee0-404a-9eb5-1964c857cb8c-kube-api-access-vjzlc\" (UniqueName: \"kubernetes.io/projected/455ae29b-fee0-404a-9eb5-1964c857cb8c-kube-api-access-vjzlc\") pod \"455ae29b-fee0-404a-9eb5-1964c857cb8c\" (UID: \"455ae29b-fee0-404a-9eb5-1964c857cb8c\") "
Oct 31 00:15:13.382805 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:13.382702    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/455ae29b-fee0-404a-9eb5-1964c857cb8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455ae29b-fee0-404a-9eb5-1964c857cb8c-plugins-config-volume\") pod \"455ae29b-fee0-404a-9eb5-1964c857cb8c\" (UID: \"455ae29b-fee0-404a-9eb5-1964c857cb8c\") "
Oct 31 00:15:13.383107 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:13.383085    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/455ae29b-fee0-404a-9eb5-1964c857cb8c-plugins-config-volume" pod "455ae29b-fee0-404a-9eb5-1964c857cb8c" (UID: "455ae29b-fee0-404a-9eb5-1964c857cb8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:13.384872 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:13.384844    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/455ae29b-fee0-404a-9eb5-1964c857cb8c-kube-api-access-vjzlc" pod "455ae29b-fee0-404a-9eb5-1964c857cb8c" (UID: "455ae29b-fee0-404a-9eb5-1964c857cb8c"). InnerVolumeSpecName "kube-api-access-vjzlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:13.437532 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:13.437502    5078 generic.go:356] "Generic (PLEG): container finished" podID="455ae29b-fee0-404a-9eb5-1964c857cb8c" containerID="29a22fb50ce9c30318485db18ad4f25c091134183b9fd29f22aa59b8ac896b37" exitCode=0
Oct 31 00:15:13.437756 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:13.437555    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7"
Oct 31 00:15:13.437756 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:13.437572    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7" event={"ID":"455ae29b-fee0-404a-9eb5-1964c857cb8c","Type":"ContainerDied","Data":"29a22fb50ce9c30318485db18ad4f25c091134183b9fd29f22aa59b8ac896b37"}
Oct 31 00:15:13.437756 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:13.437647    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7" event={"ID":"455ae29b-fee0-404a-9eb5-1964c857cb8c","Type":"ContainerDied","Data":"fc148c2a728b7b67ca2dc4bea3d0bed7787a6c6d82135385e9fa1fff62b9676f"}
Oct 31 00:15:13.437756 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:13.437672    5078 scope.go:122] "RemoveContainer" containerID="29a22fb50ce9c30318485db18ad4f25c091134183b9fd29f22aa59b8ac896b37"
Oct 31 00:15:13.446805 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:13.446787    5078 scope.go:122] "RemoveContainer" containerID="29a22fb50ce9c30318485db18ad4f25c091134183b9fd29f22aa59b8ac896b37"
Oct 31 00:15:13.447079 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:15:13.447063    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a22fb50ce9c30318485db18ad4f25c091134183b9fd29f22aa59b8ac896b37\": container with ID starting with 29a22fb50ce9c30318485db18ad4f25c091134183b9fd29f22aa59b8ac896b37 not found: ID does not exist" containerID="29a22fb50ce9c30318485db18ad4f25c091134183b9fd29f22aa59b8ac896b37"
Oct 31 00:15:13.447143 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:13.447088    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a22fb50ce9c30318485db18ad4f25c091134183b9fd29f22aa59b8ac896b37"} err="failed to get container status \"29a22fb50ce9c30318485db18ad4f25c091134183b9fd29f22aa59b8ac896b37\": rpc error: code = NotFound desc = could not find container \"29a22fb50ce9c30318485db18ad4f25c091134183b9fd29f22aa59b8ac896b37\": container with ID starting with 29a22fb50ce9c30318485db18ad4f25c091134183b9fd29f22aa59b8ac896b37 not found: ID does not exist"
Oct 31 00:15:13.471655 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:15:13.471626    5078 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod455ae29b_fee0_404a_9eb5_1964c857cb8c.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 00:15:13.483642 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:13.483622    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/455ae29b-fee0-404a-9eb5-1964c857cb8c-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:15:13.483642 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:13.483642    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjzlc\" (UniqueName: \"kubernetes.io/projected/455ae29b-fee0-404a-9eb5-1964c857cb8c-kube-api-access-vjzlc\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:15:13.601009 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:13.600976    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7"]
Oct 31 00:15:13.604176 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:13.604152    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4145bafc-cff8f8899-72tv7"]
Oct 31 00:15:14.293684 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:14.293656    5078 scope.go:122] "RemoveContainer" containerID="b9ae0aab745adf2c27106d3fe66d8e736a338bf65cab983a6f0dd6be52443ee3"
Oct 31 00:15:14.296949 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:14.296926    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="455ae29b-fee0-404a-9eb5-1964c857cb8c" path="/var/lib/kubelet/pods/455ae29b-fee0-404a-9eb5-1964c857cb8c/volumes"
Oct 31 00:15:14.801965 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:14.801931    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45"]
Oct 31 00:15:14.947001 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:14.946980    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45"
Oct 31 00:15:15.099735 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:15.099709    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c5b19ec-9d4a-4f38-907b-3e0414b7f525-kube-api-access-9c8cm\" (UniqueName: \"kubernetes.io/projected/3c5b19ec-9d4a-4f38-907b-3e0414b7f525-kube-api-access-9c8cm\") pod \"3c5b19ec-9d4a-4f38-907b-3e0414b7f525\" (UID: \"3c5b19ec-9d4a-4f38-907b-3e0414b7f525\") "
Oct 31 00:15:15.099847 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:15.099743    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c5b19ec-9d4a-4f38-907b-3e0414b7f525-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5b19ec-9d4a-4f38-907b-3e0414b7f525-maas-api-service-ca\") pod \"3c5b19ec-9d4a-4f38-907b-3e0414b7f525\" (UID: \"3c5b19ec-9d4a-4f38-907b-3e0414b7f525\") "
Oct 31 00:15:15.100136 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:15.100114    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5b19ec-9d4a-4f38-907b-3e0414b7f525-maas-api-service-ca" pod "3c5b19ec-9d4a-4f38-907b-3e0414b7f525" (UID: "3c5b19ec-9d4a-4f38-907b-3e0414b7f525"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:15.101783 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:15.101761    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5b19ec-9d4a-4f38-907b-3e0414b7f525-kube-api-access-9c8cm" pod "3c5b19ec-9d4a-4f38-907b-3e0414b7f525" (UID: "3c5b19ec-9d4a-4f38-907b-3e0414b7f525"). InnerVolumeSpecName "kube-api-access-9c8cm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:15.201038 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:15.201007    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c8cm\" (UniqueName: \"kubernetes.io/projected/3c5b19ec-9d4a-4f38-907b-3e0414b7f525-kube-api-access-9c8cm\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:15:15.201038 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:15.201035    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c5b19ec-9d4a-4f38-907b-3e0414b7f525-maas-api-service-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:15:15.450137 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:15.450061    5078 generic.go:356] "Generic (PLEG): container finished" podID="3f1e6bcb-4201-4905-980d-f7fd00f191c3" containerID="59569a38c48ec6c818c557371ef73a67ab30f092598ebb65940158c5f733f469" exitCode=60
Oct 31 00:15:15.450551 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:15.450133    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr" event={"ID":"3f1e6bcb-4201-4905-980d-f7fd00f191c3","Type":"ContainerDied","Data":"59569a38c48ec6c818c557371ef73a67ab30f092598ebb65940158c5f733f469"}
Oct 31 00:15:15.450551 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:15.450190    5078 scope.go:122] "RemoveContainer" containerID="b9ae0aab745adf2c27106d3fe66d8e736a338bf65cab983a6f0dd6be52443ee3"
Oct 31 00:15:15.450551 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:15.450525    5078 scope.go:122] "RemoveContainer" containerID="59569a38c48ec6c818c557371ef73a67ab30f092598ebb65940158c5f733f469"
Oct 31 00:15:15.450827 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:15:15.450800    5078 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890095-nc7hr_odh-ai-gateway-infra(3f1e6bcb-4201-4905-980d-f7fd00f191c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr" podUID="3f1e6bcb-4201-4905-980d-f7fd00f191c3"
Oct 31 00:15:15.451523 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:15.451507    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45"
Oct 31 00:15:15.451627 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:15.451518    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45" event={"ID":"3c5b19ec-9d4a-4f38-907b-3e0414b7f525","Type":"ContainerDied","Data":"5db607b4d63c25d087b78009ae2da618e2b7ce233c25703ac06c2e591be4987a"}
Oct 31 00:15:15.461484 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:15.461468    5078 scope.go:122] "RemoveContainer" containerID="301c5e1f74948f75509f4db77dbecdcce7a7896e7dd1b1a09036a38adb7b74ff"
Oct 31 00:15:15.672234 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:15.672201    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45"]
Oct 31 00:15:15.676115 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:15.676091    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-4145bafc-29890095-svq45"]
Oct 31 00:15:16.302777 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:16.302738    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c5b19ec-9d4a-4f38-907b-3e0414b7f525" path="/var/lib/kubelet/pods/3c5b19ec-9d4a-4f38-907b-3e0414b7f525/volumes"
Oct 31 00:15:16.576004 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:16.575899    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr"]
Oct 31 00:15:16.711101 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:16.711078    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr"
Oct 31 00:15:16.743840 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:16.743801    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx"]
Oct 31 00:15:16.745165 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:16.745147    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4145bafc-ljqxx"]
Oct 31 00:15:16.814499 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:16.814466    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f1e6bcb-4201-4905-980d-f7fd00f191c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1e6bcb-4201-4905-980d-f7fd00f191c3-maas-api-service-ca\") pod \"3f1e6bcb-4201-4905-980d-f7fd00f191c3\" (UID: \"3f1e6bcb-4201-4905-980d-f7fd00f191c3\") "
Oct 31 00:15:16.814718 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:16.814669    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f1e6bcb-4201-4905-980d-f7fd00f191c3-kube-api-access-z9n8d\" (UniqueName: \"kubernetes.io/projected/3f1e6bcb-4201-4905-980d-f7fd00f191c3-kube-api-access-z9n8d\") pod \"3f1e6bcb-4201-4905-980d-f7fd00f191c3\" (UID: \"3f1e6bcb-4201-4905-980d-f7fd00f191c3\") "
Oct 31 00:15:16.814856 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:16.814832    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f1e6bcb-4201-4905-980d-f7fd00f191c3-maas-api-service-ca" pod "3f1e6bcb-4201-4905-980d-f7fd00f191c3" (UID: "3f1e6bcb-4201-4905-980d-f7fd00f191c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:16.814954 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:16.814941    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f1e6bcb-4201-4905-980d-f7fd00f191c3-maas-api-service-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:15:16.816879 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:16.816853    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f1e6bcb-4201-4905-980d-f7fd00f191c3-kube-api-access-z9n8d" pod "3f1e6bcb-4201-4905-980d-f7fd00f191c3" (UID: "3f1e6bcb-4201-4905-980d-f7fd00f191c3"). InnerVolumeSpecName "kube-api-access-z9n8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:16.915500 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:16.915421    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9n8d\" (UniqueName: \"kubernetes.io/projected/3f1e6bcb-4201-4905-980d-f7fd00f191c3-kube-api-access-z9n8d\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:15:17.465013 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:17.464983    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr"
Oct 31 00:15:17.465231 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:17.464982    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr" event={"ID":"3f1e6bcb-4201-4905-980d-f7fd00f191c3","Type":"ContainerDied","Data":"9134ca137adb61ec6874cfa9619797a97a7ac9dd027dc9f135ce57ac8cb896f9"}
Oct 31 00:15:17.465231 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:17.465122    5078 scope.go:122] "RemoveContainer" containerID="59569a38c48ec6c818c557371ef73a67ab30f092598ebb65940158c5f733f469"
Oct 31 00:15:17.489327 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:17.489296    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"]
Oct 31 00:15:17.489506 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:17.489486    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r" podUID="7a53f2e0-cece-46a9-9ece-17c74293e761" containerName="istio-proxy" containerID="cri-o://ba70a97d60430c079b5adc6da679ee7557eb0759dde656c62fcc60439340faab" gracePeriod=30
Oct 31 00:15:17.638789 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:15:17.638723    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:15:17.742713 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:17.742631    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr"]
Oct 31 00:15:17.744902 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:17.744878    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890095-nc7hr"]
Oct 31 00:15:18.296586 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:18.296552    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f1e6bcb-4201-4905-980d-f7fd00f191c3" path="/var/lib/kubelet/pods/3f1e6bcb-4201-4905-980d-f7fd00f191c3/volumes"
Oct 31 00:15:18.296990 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:18.296977    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d473b4f3-8bd6-42fc-b6aa-62570e6deb27" path="/var/lib/kubelet/pods/d473b4f3-8bd6-42fc-b6aa-62570e6deb27/volumes"
Oct 31 00:15:19.218284 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.218249    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"]
Oct 31 00:15:19.218715 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.218570    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3f1e6bcb-4201-4905-980d-f7fd00f191c3" containerName="cleanup"
Oct 31 00:15:19.218715 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.218580    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1e6bcb-4201-4905-980d-f7fd00f191c3" containerName="cleanup"
Oct 31 00:15:19.218715 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.218589    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3c5b19ec-9d4a-4f38-907b-3e0414b7f525" containerName="cleanup"
Oct 31 00:15:19.218715 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.218607    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c5b19ec-9d4a-4f38-907b-3e0414b7f525" containerName="cleanup"
Oct 31 00:15:19.218715 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.218622    5078 cpu_manager.go:414] "containerMap: removing container" podUID="455ae29b-fee0-404a-9eb5-1964c857cb8c" containerName="payload-processing"
Oct 31 00:15:19.218715 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.218628    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="455ae29b-fee0-404a-9eb5-1964c857cb8c" containerName="payload-processing"
Oct 31 00:15:19.218715 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.218653    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3f1e6bcb-4201-4905-980d-f7fd00f191c3" containerName="cleanup"
Oct 31 00:15:19.218715 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.218657    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1e6bcb-4201-4905-980d-f7fd00f191c3" containerName="cleanup"
Oct 31 00:15:19.218715 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.218704    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1e6bcb-4201-4905-980d-f7fd00f191c3" containerName="cleanup"
Oct 31 00:15:19.218715 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.218711    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c5b19ec-9d4a-4f38-907b-3e0414b7f525" containerName="cleanup"
Oct 31 00:15:19.218715 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.218717    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1e6bcb-4201-4905-980d-f7fd00f191c3" containerName="cleanup"
Oct 31 00:15:19.218715 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.218724    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f1e6bcb-4201-4905-980d-f7fd00f191c3" containerName="cleanup"
Oct 31 00:15:19.219158 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.218729    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c5b19ec-9d4a-4f38-907b-3e0414b7f525" containerName="cleanup"
Oct 31 00:15:19.219158 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.218736    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="455ae29b-fee0-404a-9eb5-1964c857cb8c" containerName="payload-processing"
Oct 31 00:15:19.222849 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.222833    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.274972 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.274943    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-1d38f4-openshift-default-dockercfg-7grd5\""
Oct 31 00:15:19.326581 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.326550    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"]
Oct 31 00:15:19.339775 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.339707    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa64f8db-d461-41ce-be80-55c00e92c638-istio-token\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.340016 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.339944    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-workload-socket\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.340113 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.340055    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz5b6\" (UniqueName: \"kubernetes.io/projected/fa64f8db-d461-41ce-be80-55c00e92c638-kube-api-access-cz5b6\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.340177 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.340113    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa64f8db-d461-41ce-be80-55c00e92c638-istio-podinfo\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.340177 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.340144    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa64f8db-d461-41ce-be80-55c00e92c638-istiod-ca-cert\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.340292 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.340189    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-istio-data\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.340292 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.340243    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-workload-certs\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.340406 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.340331    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-credential-socket\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.340406 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.340361    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-istio-envoy\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.441327 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.441287    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-workload-socket\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.441510 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.441388    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz5b6\" (UniqueName: \"kubernetes.io/projected/fa64f8db-d461-41ce-be80-55c00e92c638-kube-api-access-cz5b6\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.441510 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.441421    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa64f8db-d461-41ce-be80-55c00e92c638-istio-podinfo\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.441510 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.441448    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa64f8db-d461-41ce-be80-55c00e92c638-istiod-ca-cert\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.441510 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.441494    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-istio-data\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.441808 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.441518    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-workload-certs\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.441808 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.441562    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-credential-socket\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.441808 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.441590    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-istio-envoy\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.441808 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.441640    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa64f8db-d461-41ce-be80-55c00e92c638-istio-token\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.441808 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.441760    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-workload-socket\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.442061 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.441997    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-istio-data\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.442204 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.442177    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-workload-certs\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.442295 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.442277    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-credential-socket\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.442531 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.442344    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa64f8db-d461-41ce-be80-55c00e92c638-istiod-ca-cert\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.444076 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.444052    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-istio-envoy\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.444282 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.444268    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa64f8db-d461-41ce-be80-55c00e92c638-istio-podinfo\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.498839 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.498783    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa64f8db-d461-41ce-be80-55c00e92c638-istio-token\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.499169 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.499145    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz5b6\" (UniqueName: \"kubernetes.io/projected/fa64f8db-d461-41ce-be80-55c00e92c638-kube-api-access-cz5b6\") pod \"e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") " pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.535841 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.535811    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:19.964807 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.964779    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"]
Oct 31 00:15:19.966649 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:15:19.966621    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa64f8db_d461_41ce_be80_55c00e92c638.slice/crio-f14840bba482fa4e470eebd8ab116cb194a598fc0c3f9abefb92eb3bbb371da8 WatchSource:0}: Error finding container f14840bba482fa4e470eebd8ab116cb194a598fc0c3f9abefb92eb3bbb371da8: Status 404 returned error can't find the container with id f14840bba482fa4e470eebd8ab116cb194a598fc0c3f9abefb92eb3bbb371da8
Oct 31 00:15:19.968915 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.968888    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:15:19.969001 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.968953    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:15:19.969001 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:19.968983    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:15:20.481499 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:20.481463    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx" event={"ID":"fa64f8db-d461-41ce-be80-55c00e92c638","Type":"ContainerStarted","Data":"38965f23267d1a3c06f1bb70a82e60471e96595cbf921cefaae881f835593115"}
Oct 31 00:15:20.481970 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:20.481522    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx" event={"ID":"fa64f8db-d461-41ce-be80-55c00e92c638","Type":"ContainerStarted","Data":"f14840bba482fa4e470eebd8ab116cb194a598fc0c3f9abefb92eb3bbb371da8"}
Oct 31 00:15:20.536803 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:20.536770    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:20.597971 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:20.597920    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx" podStartSLOduration=1.5979067740000001 podStartE2EDuration="1.597906774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:20.596308287 +0000 UTC m=+4106.484214642" watchObservedRunningTime="2026-10-31 00:15:20.597906774 +0000 UTC m=+4106.485813153"
Oct 31 00:15:21.542161 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:21.542126    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:22.490888 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.490858    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:22.491760 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.491744    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:22.752147 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.751913    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:15:22.873075 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.873044    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-token\" (UniqueName: \"kubernetes.io/projected/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-token\") pod \"7a53f2e0-cece-46a9-9ece-17c74293e761\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") "
Oct 31 00:15:22.873246 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.873102    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a53f2e0-cece-46a9-9ece-17c74293e761-kube-api-access-56tfk\" (UniqueName: \"kubernetes.io/projected/7a53f2e0-cece-46a9-9ece-17c74293e761-kube-api-access-56tfk\") pod \"7a53f2e0-cece-46a9-9ece-17c74293e761\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") "
Oct 31 00:15:22.873246 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.873148    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a53f2e0-cece-46a9-9ece-17c74293e761-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a53f2e0-cece-46a9-9ece-17c74293e761-istiod-ca-cert\") pod \"7a53f2e0-cece-46a9-9ece-17c74293e761\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") "
Oct 31 00:15:22.873246 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.873175    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-data\") pod \"7a53f2e0-cece-46a9-9ece-17c74293e761\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") "
Oct 31 00:15:22.873418 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.873334    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-credential-socket\") pod \"7a53f2e0-cece-46a9-9ece-17c74293e761\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") "
Oct 31 00:15:22.873418 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.873392    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-workload-socket\") pod \"7a53f2e0-cece-46a9-9ece-17c74293e761\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") "
Oct 31 00:15:22.873546 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.873453    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-podinfo\") pod \"7a53f2e0-cece-46a9-9ece-17c74293e761\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") "
Oct 31 00:15:22.873546 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.873474    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-data" pod "7a53f2e0-cece-46a9-9ece-17c74293e761" (UID: "7a53f2e0-cece-46a9-9ece-17c74293e761"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:15:22.873546 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.873499    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-workload-certs\") pod \"7a53f2e0-cece-46a9-9ece-17c74293e761\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") "
Oct 31 00:15:22.873765 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.873546    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-envoy\") pod \"7a53f2e0-cece-46a9-9ece-17c74293e761\" (UID: \"7a53f2e0-cece-46a9-9ece-17c74293e761\") "
Oct 31 00:15:22.873765 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.873683    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-workload-socket" pod "7a53f2e0-cece-46a9-9ece-17c74293e761" (UID: "7a53f2e0-cece-46a9-9ece-17c74293e761"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:15:22.873765 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.873590    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a53f2e0-cece-46a9-9ece-17c74293e761-istiod-ca-cert" pod "7a53f2e0-cece-46a9-9ece-17c74293e761" (UID: "7a53f2e0-cece-46a9-9ece-17c74293e761"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:15:22.873936 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.873781    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-workload-certs" pod "7a53f2e0-cece-46a9-9ece-17c74293e761" (UID: "7a53f2e0-cece-46a9-9ece-17c74293e761"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:15:22.873936 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.873845    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-credential-socket" pod "7a53f2e0-cece-46a9-9ece-17c74293e761" (UID: "7a53f2e0-cece-46a9-9ece-17c74293e761"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:15:22.874168 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.874144    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a53f2e0-cece-46a9-9ece-17c74293e761-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:15:22.874168 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.874173    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:15:22.874168 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.874189    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:15:22.874168 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.874203    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:15:22.874168 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.874218    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:15:22.875854 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.875828    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a53f2e0-cece-46a9-9ece-17c74293e761-kube-api-access-56tfk" pod "7a53f2e0-cece-46a9-9ece-17c74293e761" (UID: "7a53f2e0-cece-46a9-9ece-17c74293e761"). InnerVolumeSpecName "kube-api-access-56tfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:22.876114 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.876093    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-token" pod "7a53f2e0-cece-46a9-9ece-17c74293e761" (UID: "7a53f2e0-cece-46a9-9ece-17c74293e761"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:15:22.876196 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.876130    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-envoy" pod "7a53f2e0-cece-46a9-9ece-17c74293e761" (UID: "7a53f2e0-cece-46a9-9ece-17c74293e761"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:15:22.876243 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.876192    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-podinfo" pod "7a53f2e0-cece-46a9-9ece-17c74293e761" (UID: "7a53f2e0-cece-46a9-9ece-17c74293e761"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:15:22.975057 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.975013    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:15:22.975057 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.975049    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:15:22.975271 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.975066    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56tfk\" (UniqueName: \"kubernetes.io/projected/7a53f2e0-cece-46a9-9ece-17c74293e761-kube-api-access-56tfk\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:15:22.975271 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:22.975079    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a53f2e0-cece-46a9-9ece-17c74293e761-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:15:23.497088 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:23.497052    5078 generic.go:356] "Generic (PLEG): container finished" podID="7a53f2e0-cece-46a9-9ece-17c74293e761" containerID="ba70a97d60430c079b5adc6da679ee7557eb0759dde656c62fcc60439340faab" exitCode=0
Oct 31 00:15:23.497282 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:23.497107    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"
Oct 31 00:15:23.497282 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:23.497140    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r" event={"ID":"7a53f2e0-cece-46a9-9ece-17c74293e761","Type":"ContainerDied","Data":"ba70a97d60430c079b5adc6da679ee7557eb0759dde656c62fcc60439340faab"}
Oct 31 00:15:23.497282 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:23.497193    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r" event={"ID":"7a53f2e0-cece-46a9-9ece-17c74293e761","Type":"ContainerDied","Data":"2b8d46c5e511e5c546589be792b8a5bf7fc9e935382b50ebc8807003c5685367"}
Oct 31 00:15:23.497282 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:23.497216    5078 scope.go:122] "RemoveContainer" containerID="ba70a97d60430c079b5adc6da679ee7557eb0759dde656c62fcc60439340faab"
Oct 31 00:15:23.506975 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:23.506959    5078 scope.go:122] "RemoveContainer" containerID="88963f1eae9ac4c6aa632d320deda8196be859a2f1d2ba92ffc0df0954dcd4a3"
Oct 31 00:15:23.515160 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:23.515144    5078 scope.go:122] "RemoveContainer" containerID="ba70a97d60430c079b5adc6da679ee7557eb0759dde656c62fcc60439340faab"
Oct 31 00:15:23.515412 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:15:23.515391    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba70a97d60430c079b5adc6da679ee7557eb0759dde656c62fcc60439340faab\": container with ID starting with ba70a97d60430c079b5adc6da679ee7557eb0759dde656c62fcc60439340faab not found: ID does not exist" containerID="ba70a97d60430c079b5adc6da679ee7557eb0759dde656c62fcc60439340faab"
Oct 31 00:15:23.515469 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:23.515427    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba70a97d60430c079b5adc6da679ee7557eb0759dde656c62fcc60439340faab"} err="failed to get container status \"ba70a97d60430c079b5adc6da679ee7557eb0759dde656c62fcc60439340faab\": rpc error: code = NotFound desc = could not find container \"ba70a97d60430c079b5adc6da679ee7557eb0759dde656c62fcc60439340faab\": container with ID starting with ba70a97d60430c079b5adc6da679ee7557eb0759dde656c62fcc60439340faab not found: ID does not exist"
Oct 31 00:15:23.515469 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:23.515452    5078 scope.go:122] "RemoveContainer" containerID="88963f1eae9ac4c6aa632d320deda8196be859a2f1d2ba92ffc0df0954dcd4a3"
Oct 31 00:15:23.515801 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:15:23.515760    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88963f1eae9ac4c6aa632d320deda8196be859a2f1d2ba92ffc0df0954dcd4a3\": container with ID starting with 88963f1eae9ac4c6aa632d320deda8196be859a2f1d2ba92ffc0df0954dcd4a3 not found: ID does not exist" containerID="88963f1eae9ac4c6aa632d320deda8196be859a2f1d2ba92ffc0df0954dcd4a3"
Oct 31 00:15:23.515876 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:23.515810    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88963f1eae9ac4c6aa632d320deda8196be859a2f1d2ba92ffc0df0954dcd4a3"} err="failed to get container status \"88963f1eae9ac4c6aa632d320deda8196be859a2f1d2ba92ffc0df0954dcd4a3\": rpc error: code = NotFound desc = could not find container \"88963f1eae9ac4c6aa632d320deda8196be859a2f1d2ba92ffc0df0954dcd4a3\": container with ID starting with 88963f1eae9ac4c6aa632d320deda8196be859a2f1d2ba92ffc0df0954dcd4a3 not found: ID does not exist"
Oct 31 00:15:23.662328 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:23.662298    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"]
Oct 31 00:15:23.711989 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:23.711958    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4145bafc-openshift-default-545d555477-7k44r"]
Oct 31 00:15:24.295968 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:24.295936    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a53f2e0-cece-46a9-9ece-17c74293e761" path="/var/lib/kubelet/pods/7a53f2e0-cece-46a9-9ece-17c74293e761/volumes"
Oct 31 00:15:26.940837 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:26.940804    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r"]
Oct 31 00:15:26.941234 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:26.941119    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7a53f2e0-cece-46a9-9ece-17c74293e761" containerName="istio-proxy"
Oct 31 00:15:26.941234 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:26.941129    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a53f2e0-cece-46a9-9ece-17c74293e761" containerName="istio-proxy"
Oct 31 00:15:26.941234 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:26.941144    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3f1e6bcb-4201-4905-980d-f7fd00f191c3" containerName="cleanup"
Oct 31 00:15:26.941234 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:26.941149    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f1e6bcb-4201-4905-980d-f7fd00f191c3" containerName="cleanup"
Oct 31 00:15:26.941234 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:26.941156    5078 cpu_manager.go:414] "containerMap: removing container" podUID="3c5b19ec-9d4a-4f38-907b-3e0414b7f525" containerName="cleanup"
Oct 31 00:15:26.941234 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:26.941162    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c5b19ec-9d4a-4f38-907b-3e0414b7f525" containerName="cleanup"
Oct 31 00:15:26.941234 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:26.941217    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a53f2e0-cece-46a9-9ece-17c74293e761" containerName="istio-proxy"
Oct 31 00:15:26.941234 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:26.941227    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a53f2e0-cece-46a9-9ece-17c74293e761" containerName="istio-proxy"
Oct 31 00:15:26.945607 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:26.945578    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r"
Oct 31 00:15:27.000790 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.000749    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r"]
Oct 31 00:15:27.024064 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.024031    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xp65\" (UniqueName: \"kubernetes.io/projected/4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa-kube-api-access-8xp65\") pod \"payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r\" (UID: \"4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r"
Oct 31 00:15:27.024248 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.024146    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r\" (UID: \"4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r"
Oct 31 00:15:27.056923 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.056887    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-1d38f4\""
Oct 31 00:15:27.057120 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.057016    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-1d38f4-dockercfg-7gkrn\""
Oct 31 00:15:27.096304 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.096274    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg"]
Oct 31 00:15:27.096648 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.096636    5078 cpu_manager.go:414] "containerMap: removing container" podUID="7a53f2e0-cece-46a9-9ece-17c74293e761" containerName="istio-proxy"
Oct 31 00:15:27.096691 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.096650    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a53f2e0-cece-46a9-9ece-17c74293e761" containerName="istio-proxy"
Oct 31 00:15:27.099669 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.099646    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg"
Oct 31 00:15:27.125040 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.124997    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d65c731-e689-4855-b4ef-7f26c5cba3cf-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg\" (UID: \"6d65c731-e689-4855-b4ef-7f26c5cba3cf\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg"
Oct 31 00:15:27.125189 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.125064    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dntq8\" (UniqueName: \"kubernetes.io/projected/6d65c731-e689-4855-b4ef-7f26c5cba3cf-kube-api-access-dntq8\") pod \"payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg\" (UID: \"6d65c731-e689-4855-b4ef-7f26c5cba3cf\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg"
Oct 31 00:15:27.125189 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.125127    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xp65\" (UniqueName: \"kubernetes.io/projected/4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa-kube-api-access-8xp65\") pod \"payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r\" (UID: \"4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r"
Oct 31 00:15:27.125310 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.125213    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r\" (UID: \"4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r"
Oct 31 00:15:27.126200 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.126167    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r\" (UID: \"4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r"
Oct 31 00:15:27.155947 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.155918    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg"]
Oct 31 00:15:27.182697 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.182663    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xp65\" (UniqueName: \"kubernetes.io/projected/4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa-kube-api-access-8xp65\") pod \"payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r\" (UID: \"4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r"
Oct 31 00:15:27.225838 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.225767    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d65c731-e689-4855-b4ef-7f26c5cba3cf-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg\" (UID: \"6d65c731-e689-4855-b4ef-7f26c5cba3cf\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg"
Oct 31 00:15:27.225838 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.225800    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dntq8\" (UniqueName: \"kubernetes.io/projected/6d65c731-e689-4855-b4ef-7f26c5cba3cf-kube-api-access-dntq8\") pod \"payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg\" (UID: \"6d65c731-e689-4855-b4ef-7f26c5cba3cf\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg"
Oct 31 00:15:27.226403 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.226383    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d65c731-e689-4855-b4ef-7f26c5cba3cf-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg\" (UID: \"6d65c731-e689-4855-b4ef-7f26c5cba3cf\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg"
Oct 31 00:15:27.258633 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.258580    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r"
Oct 31 00:15:27.283061 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.283024    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dntq8\" (UniqueName: \"kubernetes.io/projected/6d65c731-e689-4855-b4ef-7f26c5cba3cf-kube-api-access-dntq8\") pod \"payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg\" (UID: \"6d65c731-e689-4855-b4ef-7f26c5cba3cf\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg"
Oct 31 00:15:27.412356 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.412322    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg"
Oct 31 00:15:27.628489 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.628461    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r"]
Oct 31 00:15:27.630935 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:15:27.630909    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a968016_dbd4_4bf8_b0f8_2b8dac4e07aa.slice/crio-8e740a976de3c88384d78427c06c6fda7409f34b267730d32d94ed9502a61cc6 WatchSource:0}: Error finding container 8e740a976de3c88384d78427c06c6fda7409f34b267730d32d94ed9502a61cc6: Status 404 returned error can't find the container with id 8e740a976de3c88384d78427c06c6fda7409f34b267730d32d94ed9502a61cc6
Oct 31 00:15:27.781931 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:27.781905    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg"]
Oct 31 00:15:27.784325 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:15:27.784300    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d65c731_e689_4855_b4ef_7f26c5cba3cf.slice/crio-5c5e01b59982558b44655309822f5b6c68c8edcf53598ec66c7c4dbe69704737 WatchSource:0}: Error finding container 5c5e01b59982558b44655309822f5b6c68c8edcf53598ec66c7c4dbe69704737: Status 404 returned error can't find the container with id 5c5e01b59982558b44655309822f5b6c68c8edcf53598ec66c7c4dbe69704737
Oct 31 00:15:28.527651 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:28.526902    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg" event={"ID":"6d65c731-e689-4855-b4ef-7f26c5cba3cf","Type":"ContainerStarted","Data":"f7ad56d99f89979e01930fabab58cc31bd0338a19284f55bafa9a16a06e9d0e4"}
Oct 31 00:15:28.527651 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:28.526979    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg" event={"ID":"6d65c731-e689-4855-b4ef-7f26c5cba3cf","Type":"ContainerStarted","Data":"5c5e01b59982558b44655309822f5b6c68c8edcf53598ec66c7c4dbe69704737"}
Oct 31 00:15:28.527651 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:28.527052    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg"
Oct 31 00:15:28.529345 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:28.529318    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r" event={"ID":"4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa","Type":"ContainerStarted","Data":"a853dc9c68c87f52eb25c6971ac6f1135cd0d35508c4dde71e727790e59fd884"}
Oct 31 00:15:28.529479 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:28.529366    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r" event={"ID":"4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa","Type":"ContainerStarted","Data":"8e740a976de3c88384d78427c06c6fda7409f34b267730d32d94ed9502a61cc6"}
Oct 31 00:15:28.529479 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:28.529397    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r"
Oct 31 00:15:28.639676 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:28.639616    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg" podStartSLOduration=1.639577649 podStartE2EDuration="1.639577649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:28.63865373 +0000 UTC m=+4114.526560085" watchObservedRunningTime="2026-10-31 00:15:28.639577649 +0000 UTC m=+4114.527484004"
Oct 31 00:15:28.752166 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:28.752119    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r" podStartSLOduration=2.752107325 podStartE2EDuration="2.752107325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:28.750728483 +0000 UTC m=+4114.638634836" watchObservedRunningTime="2026-10-31 00:15:28.752107325 +0000 UTC m=+4114.640013671"
Oct 31 00:15:39.554637 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:39.548751    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r"
Oct 31 00:15:39.554637 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:39.550872    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg"
Oct 31 00:15:42.634821 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:42.634791    5078 generic.go:356] "Generic (PLEG): container finished" podID="fa64f8db-d461-41ce-be80-55c00e92c638" containerID="38965f23267d1a3c06f1bb70a82e60471e96595cbf921cefaae881f835593115" exitCode=0
Oct 31 00:15:42.635311 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:42.634842    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx" event={"ID":"fa64f8db-d461-41ce-be80-55c00e92c638","Type":"ContainerDied","Data":"38965f23267d1a3c06f1bb70a82e60471e96595cbf921cefaae881f835593115"}
Oct 31 00:15:42.635311 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:42.635194    5078 scope.go:122] "RemoveContainer" containerID="38965f23267d1a3c06f1bb70a82e60471e96595cbf921cefaae881f835593115"
Oct 31 00:15:42.636362 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:42.636330    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:15:42.636452 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:42.636404    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:15:42.636452 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:42.636440    5078 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Oct 31 00:15:43.642195 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:43.642162    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx" event={"ID":"fa64f8db-d461-41ce-be80-55c00e92c638","Type":"ContainerStarted","Data":"f6caa76594154b53914386f2f1ec547e87fa735a0f45c2adef1c3da920445de3"}
Oct 31 00:15:44.536350 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:44.536313    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:44.536549 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:44.536489    5078 patch_prober.go:29] interesting pod/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 31 00:15:44.536694 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:44.536558    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx" podUID="fa64f8db-d461-41ce-be80-55c00e92c638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 31 00:15:45.536460 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:45.536427    5078 patch_prober.go:29] interesting pod/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 31 00:15:45.536897 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:45.536480    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx" podUID="fa64f8db-d461-41ce-be80-55c00e92c638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 31 00:15:46.536854 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:46.536815    5078 patch_prober.go:29] interesting pod/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 31 00:15:46.537338 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:46.536865    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx" podUID="fa64f8db-d461-41ce-be80-55c00e92c638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 31 00:15:47.536549 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:47.536510    5078 patch_prober.go:29] interesting pod/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 31 00:15:47.536764 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:47.536562    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx" podUID="fa64f8db-d461-41ce-be80-55c00e92c638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 31 00:15:47.607614 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:15:47.607569    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:15:48.536955 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:48.536923    5078 patch_prober.go:29] interesting pod/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused" start-of-body=
Oct 31 00:15:48.537189 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:48.536969    5078 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx" podUID="fa64f8db-d461-41ce-be80-55c00e92c638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.43:15021/healthz/ready\": dial tcp 10.133.1.43:15021: connect: connection refused"
Oct 31 00:15:49.540864 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:49.540831    5078 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:49.541343 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:49.541109    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:15:49.541830 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:15:49.541807    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:16:04.566296 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:16:04.566225    5078 scope.go:122] "RemoveContainer" containerID="d8b7eb70953856104d2da949292c4e3c24a151034077235a144b0b564fd98629"
Oct 31 00:16:04.576979 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:16:04.576958    5078 scope.go:122] "RemoveContainer" containerID="b0848c3431fab86b1bcbeee86eadd338d4147c56fb75f3e8782dc3d08c9da016"
Oct 31 00:16:17.050573 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:16:17.050543    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls"]
Oct 31 00:16:17.100163 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:16:17.100121    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6e34b49-fbmls"]
Oct 31 00:16:17.589747 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:16:17.589715    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:16:18.296576 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:16:18.296539    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="534d2089-ffd2-46dd-9fc5-01851c312a6b" path="/var/lib/kubelet/pods/534d2089-ffd2-46dd-9fc5-01851c312a6b/volumes"
Oct 31 00:16:47.589866 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:16:47.589819    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:17:03.157757 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:03.157725    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx"]
Oct 31 00:17:03.161089 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:03.161072    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx"
Oct 31 00:17:03.216438 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:03.216405    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:17:03.265221 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:03.265178    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnrc9\""
Oct 31 00:17:03.265221 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:03.265195    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:17:03.265221 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:03.265189    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:17:03.266868 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:03.266841    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx"]
Oct 31 00:17:03.308867 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:03.308836    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx\" (UID: \"ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx"
Oct 31 00:17:03.309022 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:03.308947    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx\" (UID: \"ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx"
Oct 31 00:17:03.309022 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:03.308998    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t86zt\" (UniqueName: \"kubernetes.io/projected/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-kube-api-access-t86zt\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx\" (UID: \"ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx"
Oct 31 00:17:03.410448 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:03.410382    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx\" (UID: \"ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx"
Oct 31 00:17:03.410448 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:03.410439    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx\" (UID: \"ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx"
Oct 31 00:17:03.410663 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:03.410478    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t86zt\" (UniqueName: \"kubernetes.io/projected/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-kube-api-access-t86zt\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx\" (UID: \"ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx"
Oct 31 00:17:03.411127 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:03.411104    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx\" (UID: \"ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx"
Oct 31 00:17:03.412867 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:03.412847    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx\" (UID: \"ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx"
Oct 31 00:17:03.472894 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:03.472870    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t86zt\" (UniqueName: \"kubernetes.io/projected/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-kube-api-access-t86zt\") pod \"maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx\" (UID: \"ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx"
Oct 31 00:17:03.473023 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:03.472981    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx"
Oct 31 00:17:03.841550 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:03.841342    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx"]
Oct 31 00:17:03.844272 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:17:03.844243    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad72a40b_01d3_4fc3_9a3a_dfbdf3ed9c99.slice/crio-ebea86c70ee96531a42d7268c080bd1b18049da7e06c7faab4b1bc322f17acc5 WatchSource:0}: Error finding container ebea86c70ee96531a42d7268c080bd1b18049da7e06c7faab4b1bc322f17acc5: Status 404 returned error can't find the container with id ebea86c70ee96531a42d7268c080bd1b18049da7e06c7faab4b1bc322f17acc5
Oct 31 00:17:04.049662 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:04.049629    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx" event={"ID":"ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99","Type":"ContainerStarted","Data":"ebea86c70ee96531a42d7268c080bd1b18049da7e06c7faab4b1bc322f17acc5"}
Oct 31 00:17:05.055758 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:05.055728    5078 generic.go:356] "Generic (PLEG): container finished" podID="ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99" containerID="2d7c4cc48f3d635c970c15cba5c737f131f3398b5147ee5a2cb0a79334e6930b" exitCode=0
Oct 31 00:17:05.056198 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:05.055817    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx" event={"ID":"ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99","Type":"ContainerDied","Data":"2d7c4cc48f3d635c970c15cba5c737f131f3398b5147ee5a2cb0a79334e6930b"}
Oct 31 00:17:06.190672 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:06.190643    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx"
Oct 31 00:17:06.235340 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:06.235313    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-maas-api-xks-ca\") pod \"ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99\" (UID: \"ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99\") "
Oct 31 00:17:06.235473 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:06.235388    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-kube-api-access-t86zt\" (UniqueName: \"kubernetes.io/projected/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-kube-api-access-t86zt\") pod \"ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99\" (UID: \"ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99\") "
Oct 31 00:17:06.235473 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:06.235452    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-maas-api-openshift-ca\") pod \"ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99\" (UID: \"ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99\") "
Oct 31 00:17:06.235839 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:06.235815    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-maas-api-openshift-ca" pod "ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99" (UID: "ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:06.237535 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:06.237513    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-maas-api-xks-ca" pod "ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99" (UID: "ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:06.237627 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:06.237569    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-kube-api-access-t86zt" pod "ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99" (UID: "ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99"). InnerVolumeSpecName "kube-api-access-t86zt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:06.336936 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:06.336855    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t86zt\" (UniqueName: \"kubernetes.io/projected/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-kube-api-access-t86zt\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:06.336936 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:06.336894    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:06.336936 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:06.336911    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:07.065856 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:07.065820    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx" event={"ID":"ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99","Type":"ContainerDied","Data":"ebea86c70ee96531a42d7268c080bd1b18049da7e06c7faab4b1bc322f17acc5"}
Oct 31 00:17:07.066049 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:07.065875    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebea86c70ee96531a42d7268c080bd1b18049da7e06c7faab4b1bc322f17acc5"
Oct 31 00:17:07.066049 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:07.065833    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx"
Oct 31 00:17:14.692559 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:14.692523    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg"]
Oct 31 00:17:14.801555 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:14.801524    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r"]
Oct 31 00:17:17.590348 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:17:17.590314    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:17:19.693686 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:19.693637    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg" podUID="6d65c731-e689-4855-b4ef-7f26c5cba3cf" containerName="payload-processing" containerID="cri-o://f7ad56d99f89979e01930fabab58cc31bd0338a19284f55bafa9a16a06e9d0e4" gracePeriod=25
Oct 31 00:17:19.802895 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:19.802844    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r" podUID="4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa" containerName="payload-pre-processing" containerID="cri-o://a853dc9c68c87f52eb25c6971ac6f1135cd0d35508c4dde71e727790e59fd884" gracePeriod=25
Oct 31 00:17:19.935923 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:19.935862    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg"
Oct 31 00:17:20.036699 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.036677    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r"
Oct 31 00:17:20.063591 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.063567    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d65c731-e689-4855-b4ef-7f26c5cba3cf-kube-api-access-dntq8\" (UniqueName: \"kubernetes.io/projected/6d65c731-e689-4855-b4ef-7f26c5cba3cf-kube-api-access-dntq8\") pod \"6d65c731-e689-4855-b4ef-7f26c5cba3cf\" (UID: \"6d65c731-e689-4855-b4ef-7f26c5cba3cf\") "
Oct 31 00:17:20.063755 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.063715    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d65c731-e689-4855-b4ef-7f26c5cba3cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d65c731-e689-4855-b4ef-7f26c5cba3cf-plugins-config-volume\") pod \"6d65c731-e689-4855-b4ef-7f26c5cba3cf\" (UID: \"6d65c731-e689-4855-b4ef-7f26c5cba3cf\") "
Oct 31 00:17:20.064072 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.064051    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d65c731-e689-4855-b4ef-7f26c5cba3cf-plugins-config-volume" pod "6d65c731-e689-4855-b4ef-7f26c5cba3cf" (UID: "6d65c731-e689-4855-b4ef-7f26c5cba3cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:20.065840 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.065819    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d65c731-e689-4855-b4ef-7f26c5cba3cf-kube-api-access-dntq8" pod "6d65c731-e689-4855-b4ef-7f26c5cba3cf" (UID: "6d65c731-e689-4855-b4ef-7f26c5cba3cf"). InnerVolumeSpecName "kube-api-access-dntq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:20.126700 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.126671    5078 generic.go:356] "Generic (PLEG): container finished" podID="6d65c731-e689-4855-b4ef-7f26c5cba3cf" containerID="f7ad56d99f89979e01930fabab58cc31bd0338a19284f55bafa9a16a06e9d0e4" exitCode=0
Oct 31 00:17:20.126832 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.126723    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg"
Oct 31 00:17:20.126832 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.126763    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg" event={"ID":"6d65c731-e689-4855-b4ef-7f26c5cba3cf","Type":"ContainerDied","Data":"f7ad56d99f89979e01930fabab58cc31bd0338a19284f55bafa9a16a06e9d0e4"}
Oct 31 00:17:20.126967 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.126866    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg" event={"ID":"6d65c731-e689-4855-b4ef-7f26c5cba3cf","Type":"ContainerDied","Data":"5c5e01b59982558b44655309822f5b6c68c8edcf53598ec66c7c4dbe69704737"}
Oct 31 00:17:20.126967 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.126903    5078 scope.go:122] "RemoveContainer" containerID="f7ad56d99f89979e01930fabab58cc31bd0338a19284f55bafa9a16a06e9d0e4"
Oct 31 00:17:20.128120 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.128099    5078 generic.go:356] "Generic (PLEG): container finished" podID="4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa" containerID="a853dc9c68c87f52eb25c6971ac6f1135cd0d35508c4dde71e727790e59fd884" exitCode=0
Oct 31 00:17:20.128206 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.128136    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r"
Oct 31 00:17:20.128206 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.128174    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r" event={"ID":"4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa","Type":"ContainerDied","Data":"a853dc9c68c87f52eb25c6971ac6f1135cd0d35508c4dde71e727790e59fd884"}
Oct 31 00:17:20.128343 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.128217    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r" event={"ID":"4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa","Type":"ContainerDied","Data":"8e740a976de3c88384d78427c06c6fda7409f34b267730d32d94ed9502a61cc6"}
Oct 31 00:17:20.135720 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.135704    5078 scope.go:122] "RemoveContainer" containerID="f7ad56d99f89979e01930fabab58cc31bd0338a19284f55bafa9a16a06e9d0e4"
Oct 31 00:17:20.136003 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:17:20.135984    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7ad56d99f89979e01930fabab58cc31bd0338a19284f55bafa9a16a06e9d0e4\": container with ID starting with f7ad56d99f89979e01930fabab58cc31bd0338a19284f55bafa9a16a06e9d0e4 not found: ID does not exist" containerID="f7ad56d99f89979e01930fabab58cc31bd0338a19284f55bafa9a16a06e9d0e4"
Oct 31 00:17:20.136058 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.136010    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7ad56d99f89979e01930fabab58cc31bd0338a19284f55bafa9a16a06e9d0e4"} err="failed to get container status \"f7ad56d99f89979e01930fabab58cc31bd0338a19284f55bafa9a16a06e9d0e4\": rpc error: code = NotFound desc = could not find container \"f7ad56d99f89979e01930fabab58cc31bd0338a19284f55bafa9a16a06e9d0e4\": container with ID starting with f7ad56d99f89979e01930fabab58cc31bd0338a19284f55bafa9a16a06e9d0e4 not found: ID does not exist"
Oct 31 00:17:20.136058 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.136026    5078 scope.go:122] "RemoveContainer" containerID="a853dc9c68c87f52eb25c6971ac6f1135cd0d35508c4dde71e727790e59fd884"
Oct 31 00:17:20.143280 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.143265    5078 scope.go:122] "RemoveContainer" containerID="a853dc9c68c87f52eb25c6971ac6f1135cd0d35508c4dde71e727790e59fd884"
Oct 31 00:17:20.143549 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:17:20.143524    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a853dc9c68c87f52eb25c6971ac6f1135cd0d35508c4dde71e727790e59fd884\": container with ID starting with a853dc9c68c87f52eb25c6971ac6f1135cd0d35508c4dde71e727790e59fd884 not found: ID does not exist" containerID="a853dc9c68c87f52eb25c6971ac6f1135cd0d35508c4dde71e727790e59fd884"
Oct 31 00:17:20.143629 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.143560    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a853dc9c68c87f52eb25c6971ac6f1135cd0d35508c4dde71e727790e59fd884"} err="failed to get container status \"a853dc9c68c87f52eb25c6971ac6f1135cd0d35508c4dde71e727790e59fd884\": rpc error: code = NotFound desc = could not find container \"a853dc9c68c87f52eb25c6971ac6f1135cd0d35508c4dde71e727790e59fd884\": container with ID starting with a853dc9c68c87f52eb25c6971ac6f1135cd0d35508c4dde71e727790e59fd884 not found: ID does not exist"
Oct 31 00:17:20.164218 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.164196    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa-kube-api-access-8xp65\" (UniqueName: \"kubernetes.io/projected/4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa-kube-api-access-8xp65\") pod \"4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa\" (UID: \"4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa\") "
Oct 31 00:17:20.164340 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.164287    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa-plugins-config-volume\") pod \"4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa\" (UID: \"4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa\") "
Oct 31 00:17:20.164702 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.164579    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d65c731-e689-4855-b4ef-7f26c5cba3cf-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:20.164702 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.164641    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dntq8\" (UniqueName: \"kubernetes.io/projected/6d65c731-e689-4855-b4ef-7f26c5cba3cf-kube-api-access-dntq8\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:20.165039 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.165014    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa-plugins-config-volume" pod "4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa" (UID: "4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:20.166740 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.166720    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa-kube-api-access-8xp65" pod "4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa" (UID: "4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa"). InnerVolumeSpecName "kube-api-access-8xp65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:20.265652 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.265560    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xp65\" (UniqueName: \"kubernetes.io/projected/4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa-kube-api-access-8xp65\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:20.265652 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.265608    5078 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa-plugins-config-volume\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:20.274065 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.274040    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg"]
Oct 31 00:17:20.275428 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.275408    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-1d38f4-7bc6bfb49f-fndwg"]
Oct 31 00:17:20.296185 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.296157    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d65c731-e689-4855-b4ef-7f26c5cba3cf" path="/var/lib/kubelet/pods/6d65c731-e689-4855-b4ef-7f26c5cba3cf/volumes"
Oct 31 00:17:20.542020 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.541980    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r"]
Oct 31 00:17:20.543814 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:20.543795    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-1d38f4-7fdf4f96c9-z5h7r"]
Oct 31 00:17:22.296448 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:22.296419    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa" path="/var/lib/kubelet/pods/4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa/volumes"
Oct 31 00:17:30.045172 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:30.045136    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx"]
Oct 31 00:17:30.094661 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:30.094631    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-1d38f4-9xqjx"]
Oct 31 00:17:30.296230 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:30.296156    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99" path="/var/lib/kubelet/pods/ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99/volumes"
Oct 31 00:17:36.941743 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:36.941655    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"]
Oct 31 00:17:36.942141 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:36.941855    5078 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx" podUID="fa64f8db-d461-41ce-be80-55c00e92c638" containerName="istio-proxy" containerID="cri-o://f6caa76594154b53914386f2f1ec547e87fa735a0f45c2adef1c3da920445de3" gracePeriod=30
Oct 31 00:17:42.183865 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.183843    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:17:42.238410 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.238335    5078 generic.go:356] "Generic (PLEG): container finished" podID="fa64f8db-d461-41ce-be80-55c00e92c638" containerID="f6caa76594154b53914386f2f1ec547e87fa735a0f45c2adef1c3da920445de3" exitCode=0
Oct 31 00:17:42.238410 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.238380    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"
Oct 31 00:17:42.238410 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.238404    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx" event={"ID":"fa64f8db-d461-41ce-be80-55c00e92c638","Type":"ContainerDied","Data":"f6caa76594154b53914386f2f1ec547e87fa735a0f45c2adef1c3da920445de3"}
Oct 31 00:17:42.238688 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.238493    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx" event={"ID":"fa64f8db-d461-41ce-be80-55c00e92c638","Type":"ContainerDied","Data":"f14840bba482fa4e470eebd8ab116cb194a598fc0c3f9abefb92eb3bbb371da8"}
Oct 31 00:17:42.238688 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.238525    5078 scope.go:122] "RemoveContainer" containerID="f6caa76594154b53914386f2f1ec547e87fa735a0f45c2adef1c3da920445de3"
Oct 31 00:17:42.246884 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.246860    5078 scope.go:122] "RemoveContainer" containerID="38965f23267d1a3c06f1bb70a82e60471e96595cbf921cefaae881f835593115"
Oct 31 00:17:42.254470 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.254450    5078 scope.go:122] "RemoveContainer" containerID="f6caa76594154b53914386f2f1ec547e87fa735a0f45c2adef1c3da920445de3"
Oct 31 00:17:42.254746 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:17:42.254725    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6caa76594154b53914386f2f1ec547e87fa735a0f45c2adef1c3da920445de3\": container with ID starting with f6caa76594154b53914386f2f1ec547e87fa735a0f45c2adef1c3da920445de3 not found: ID does not exist" containerID="f6caa76594154b53914386f2f1ec547e87fa735a0f45c2adef1c3da920445de3"
Oct 31 00:17:42.254796 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.254756    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6caa76594154b53914386f2f1ec547e87fa735a0f45c2adef1c3da920445de3"} err="failed to get container status \"f6caa76594154b53914386f2f1ec547e87fa735a0f45c2adef1c3da920445de3\": rpc error: code = NotFound desc = could not find container \"f6caa76594154b53914386f2f1ec547e87fa735a0f45c2adef1c3da920445de3\": container with ID starting with f6caa76594154b53914386f2f1ec547e87fa735a0f45c2adef1c3da920445de3 not found: ID does not exist"
Oct 31 00:17:42.254796 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.254774    5078 scope.go:122] "RemoveContainer" containerID="38965f23267d1a3c06f1bb70a82e60471e96595cbf921cefaae881f835593115"
Oct 31 00:17:42.255012 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:17:42.254993    5078 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38965f23267d1a3c06f1bb70a82e60471e96595cbf921cefaae881f835593115\": container with ID starting with 38965f23267d1a3c06f1bb70a82e60471e96595cbf921cefaae881f835593115 not found: ID does not exist" containerID="38965f23267d1a3c06f1bb70a82e60471e96595cbf921cefaae881f835593115"
Oct 31 00:17:42.255056 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.255019    5078 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38965f23267d1a3c06f1bb70a82e60471e96595cbf921cefaae881f835593115"} err="failed to get container status \"38965f23267d1a3c06f1bb70a82e60471e96595cbf921cefaae881f835593115\": rpc error: code = NotFound desc = could not find container \"38965f23267d1a3c06f1bb70a82e60471e96595cbf921cefaae881f835593115\": container with ID starting with 38965f23267d1a3c06f1bb70a82e60471e96595cbf921cefaae881f835593115 not found: ID does not exist"
Oct 31 00:17:42.279533 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.279510    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-workload-certs\") pod \"fa64f8db-d461-41ce-be80-55c00e92c638\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") "
Oct 31 00:17:42.279638 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.279539    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-workload-socket\") pod \"fa64f8db-d461-41ce-be80-55c00e92c638\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") "
Oct 31 00:17:42.279638 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.279588    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-istio-data\") pod \"fa64f8db-d461-41ce-be80-55c00e92c638\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") "
Oct 31 00:17:42.279731 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.279698    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa64f8db-d461-41ce-be80-55c00e92c638-istio-token\" (UniqueName: \"kubernetes.io/projected/fa64f8db-d461-41ce-be80-55c00e92c638-istio-token\") pod \"fa64f8db-d461-41ce-be80-55c00e92c638\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") "
Oct 31 00:17:42.279782 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.279741    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa64f8db-d461-41ce-be80-55c00e92c638-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa64f8db-d461-41ce-be80-55c00e92c638-istiod-ca-cert\") pod \"fa64f8db-d461-41ce-be80-55c00e92c638\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") "
Oct 31 00:17:42.279822 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.279790    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa64f8db-d461-41ce-be80-55c00e92c638-kube-api-access-cz5b6\" (UniqueName: \"kubernetes.io/projected/fa64f8db-d461-41ce-be80-55c00e92c638-kube-api-access-cz5b6\") pod \"fa64f8db-d461-41ce-be80-55c00e92c638\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") "
Oct 31 00:17:42.279867 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.279824    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-istio-envoy\") pod \"fa64f8db-d461-41ce-be80-55c00e92c638\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") "
Oct 31 00:17:42.279867 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.279828    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-workload-certs" pod "fa64f8db-d461-41ce-be80-55c00e92c638" (UID: "fa64f8db-d461-41ce-be80-55c00e92c638"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:42.279958 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.279861    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-workload-socket" pod "fa64f8db-d461-41ce-be80-55c00e92c638" (UID: "fa64f8db-d461-41ce-be80-55c00e92c638"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:42.279958 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.279868    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa64f8db-d461-41ce-be80-55c00e92c638-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa64f8db-d461-41ce-be80-55c00e92c638-istio-podinfo\") pod \"fa64f8db-d461-41ce-be80-55c00e92c638\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") "
Oct 31 00:17:42.279958 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.279914    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-istio-data" pod "fa64f8db-d461-41ce-be80-55c00e92c638" (UID: "fa64f8db-d461-41ce-be80-55c00e92c638"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:42.280218 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.280195    5078 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-workload-certs\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:42.280314 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.280225    5078 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-workload-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:42.280314 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.280244    5078 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-istio-data\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:42.280314 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.280215    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa64f8db-d461-41ce-be80-55c00e92c638-istiod-ca-cert" pod "fa64f8db-d461-41ce-be80-55c00e92c638" (UID: "fa64f8db-d461-41ce-be80-55c00e92c638"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:42.281933 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.281909    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa64f8db-d461-41ce-be80-55c00e92c638-istio-token" pod "fa64f8db-d461-41ce-be80-55c00e92c638" (UID: "fa64f8db-d461-41ce-be80-55c00e92c638"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:42.282033 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.282012    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-istio-envoy" pod "fa64f8db-d461-41ce-be80-55c00e92c638" (UID: "fa64f8db-d461-41ce-be80-55c00e92c638"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:42.282513 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.282494    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa64f8db-d461-41ce-be80-55c00e92c638-istio-podinfo" pod "fa64f8db-d461-41ce-be80-55c00e92c638" (UID: "fa64f8db-d461-41ce-be80-55c00e92c638"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:17:42.282513 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.282500    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa64f8db-d461-41ce-be80-55c00e92c638-kube-api-access-cz5b6" pod "fa64f8db-d461-41ce-be80-55c00e92c638" (UID: "fa64f8db-d461-41ce-be80-55c00e92c638"). InnerVolumeSpecName "kube-api-access-cz5b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:42.381161 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.381130    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-credential-socket\") pod \"fa64f8db-d461-41ce-be80-55c00e92c638\" (UID: \"fa64f8db-d461-41ce-be80-55c00e92c638\") "
Oct 31 00:17:42.381400 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.381369    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-credential-socket" pod "fa64f8db-d461-41ce-be80-55c00e92c638" (UID: "fa64f8db-d461-41ce-be80-55c00e92c638"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:42.381579 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.381547    5078 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-credential-socket\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:42.381579 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.381573    5078 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa64f8db-d461-41ce-be80-55c00e92c638-istio-token\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:42.381808 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.381590    5078 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa64f8db-d461-41ce-be80-55c00e92c638-istiod-ca-cert\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:42.381808 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.381625    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz5b6\" (UniqueName: \"kubernetes.io/projected/fa64f8db-d461-41ce-be80-55c00e92c638-kube-api-access-cz5b6\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:42.381808 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.381642    5078 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa64f8db-d461-41ce-be80-55c00e92c638-istio-envoy\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:42.381808 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.381655    5078 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa64f8db-d461-41ce-be80-55c00e92c638-istio-podinfo\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:42.661656 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.661616    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"]
Oct 31 00:17:42.665723 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:42.665699    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-1d38f4-openshift-default-7656bc6c89-t8ngx"]
Oct 31 00:17:44.295567 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:17:44.295526    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa64f8db-d461-41ce-be80-55c00e92c638" path="/var/lib/kubelet/pods/fa64f8db-d461-41ce-be80-55c00e92c638/volumes"
Oct 31 00:17:47.669010 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:17:47.666022    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:18:04.331276 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.331235    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d"]
Oct 31 00:18:04.331913 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.331702    5078 cpu_manager.go:414] "containerMap: removing container" podUID="6d65c731-e689-4855-b4ef-7f26c5cba3cf" containerName="payload-processing"
Oct 31 00:18:04.331913 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.331721    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d65c731-e689-4855-b4ef-7f26c5cba3cf" containerName="payload-processing"
Oct 31 00:18:04.331913 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.331735    5078 cpu_manager.go:414] "containerMap: removing container" podUID="fa64f8db-d461-41ce-be80-55c00e92c638" containerName="istio-proxy"
Oct 31 00:18:04.331913 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.331742    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa64f8db-d461-41ce-be80-55c00e92c638" containerName="istio-proxy"
Oct 31 00:18:04.331913 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.331757    5078 cpu_manager.go:414] "containerMap: removing container" podUID="4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa" containerName="payload-pre-processing"
Oct 31 00:18:04.331913 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.331764    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa" containerName="payload-pre-processing"
Oct 31 00:18:04.331913 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.331793    5078 cpu_manager.go:414] "containerMap: removing container" podUID="ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99" containerName="revoke-keys"
Oct 31 00:18:04.331913 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.331800    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99" containerName="revoke-keys"
Oct 31 00:18:04.331913 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.331877    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad72a40b-01d3-4fc3-9a3a-dfbdf3ed9c99" containerName="revoke-keys"
Oct 31 00:18:04.331913 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.331889    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa64f8db-d461-41ce-be80-55c00e92c638" containerName="istio-proxy"
Oct 31 00:18:04.331913 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.331902    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d65c731-e689-4855-b4ef-7f26c5cba3cf" containerName="payload-processing"
Oct 31 00:18:04.331913 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.331911    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa64f8db-d461-41ce-be80-55c00e92c638" containerName="istio-proxy"
Oct 31 00:18:04.331913 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.331920    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a968016-dbd4-4bf8-b0f8-2b8dac4e07aa" containerName="payload-pre-processing"
Oct 31 00:18:04.334922 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.334902    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d"
Oct 31 00:18:04.434757 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.434724    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-pxw2d\" (UID: \"1ee8f68b-fe59-4410-b1f6-861f71bffaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d"
Oct 31 00:18:04.434929 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.434858    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvjtl\" (UniqueName: \"kubernetes.io/projected/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-kube-api-access-tvjtl\") pod \"maas-api-revoke-sub-f03629db-pxw2d\" (UID: \"1ee8f68b-fe59-4410-b1f6-861f71bffaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d"
Oct 31 00:18:04.434929 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.434912    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-pxw2d\" (UID: \"1ee8f68b-fe59-4410-b1f6-861f71bffaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d"
Oct 31 00:18:04.438557 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.438529    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:18:04.438557 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.438542    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:18:04.438557 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.438535    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:18:04.438816 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.438692    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnrc9\""
Oct 31 00:18:04.440167 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.440147    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d"]
Oct 31 00:18:04.535960 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.535919    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-pxw2d\" (UID: \"1ee8f68b-fe59-4410-b1f6-861f71bffaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d"
Oct 31 00:18:04.536152 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.535966    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-pxw2d\" (UID: \"1ee8f68b-fe59-4410-b1f6-861f71bffaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d"
Oct 31 00:18:04.536152 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.536070    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvjtl\" (UniqueName: \"kubernetes.io/projected/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-kube-api-access-tvjtl\") pod \"maas-api-revoke-sub-f03629db-pxw2d\" (UID: \"1ee8f68b-fe59-4410-b1f6-861f71bffaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d"
Oct 31 00:18:04.536657 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.536633    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-pxw2d\" (UID: \"1ee8f68b-fe59-4410-b1f6-861f71bffaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d"
Oct 31 00:18:04.538441 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.538421    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-pxw2d\" (UID: \"1ee8f68b-fe59-4410-b1f6-861f71bffaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d"
Oct 31 00:18:04.592639 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.592579    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvjtl\" (UniqueName: \"kubernetes.io/projected/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-kube-api-access-tvjtl\") pod \"maas-api-revoke-sub-f03629db-pxw2d\" (UID: \"1ee8f68b-fe59-4410-b1f6-861f71bffaf7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d"
Oct 31 00:18:04.619518 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.619487    5078 scope.go:122] "RemoveContainer" containerID="84a46391484248f7507218ae917a3b11524fb1ba31873fbc5292c0fbcda6b5fd"
Oct 31 00:18:04.629102 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.629084    5078 scope.go:122] "RemoveContainer" containerID="4dc0b376e2af9fd44da77da2a4adc084656c45f2619fd36ceafeec1e22613479"
Oct 31 00:18:04.649286 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:04.649264    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d"
Oct 31 00:18:05.019911 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:05.019834    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d"]
Oct 31 00:18:05.022755 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:18:05.022726    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee8f68b_fe59_4410_b1f6_861f71bffaf7.slice/crio-7c588e5b9e1d115e963df220d94d63ddb015e57bd8c83886321b67d3300d00d7 WatchSource:0}: Error finding container 7c588e5b9e1d115e963df220d94d63ddb015e57bd8c83886321b67d3300d00d7: Status 404 returned error can't find the container with id 7c588e5b9e1d115e963df220d94d63ddb015e57bd8c83886321b67d3300d00d7
Oct 31 00:18:05.384649 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:05.384618    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d" event={"ID":"1ee8f68b-fe59-4410-b1f6-861f71bffaf7","Type":"ContainerStarted","Data":"264bff134a176b0897c293289857135f2525be25ee438e27e6a0d4d636be1df6"}
Oct 31 00:18:05.385135 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:05.384672    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d" event={"ID":"1ee8f68b-fe59-4410-b1f6-861f71bffaf7","Type":"ContainerStarted","Data":"7c588e5b9e1d115e963df220d94d63ddb015e57bd8c83886321b67d3300d00d7"}
Oct 31 00:18:05.499033 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:05.498290    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d" podStartSLOduration=1.498271371 podStartE2EDuration="1.498271371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:05.497330746 +0000 UTC m=+4271.385237096" watchObservedRunningTime="2026-10-31 00:18:05.498271371 +0000 UTC m=+4271.386177726"
Oct 31 00:18:06.389649 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:06.389612    5078 generic.go:356] "Generic (PLEG): container finished" podID="1ee8f68b-fe59-4410-b1f6-861f71bffaf7" containerID="264bff134a176b0897c293289857135f2525be25ee438e27e6a0d4d636be1df6" exitCode=0
Oct 31 00:18:06.390051 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:06.389686    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d" event={"ID":"1ee8f68b-fe59-4410-b1f6-861f71bffaf7","Type":"ContainerDied","Data":"264bff134a176b0897c293289857135f2525be25ee438e27e6a0d4d636be1df6"}
Oct 31 00:18:07.524036 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:07.524011    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d"
Oct 31 00:18:07.667117 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:07.667037    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-maas-api-openshift-ca\") pod \"1ee8f68b-fe59-4410-b1f6-861f71bffaf7\" (UID: \"1ee8f68b-fe59-4410-b1f6-861f71bffaf7\") "
Oct 31 00:18:07.667298 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:07.667147    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-kube-api-access-tvjtl\" (UniqueName: \"kubernetes.io/projected/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-kube-api-access-tvjtl\") pod \"1ee8f68b-fe59-4410-b1f6-861f71bffaf7\" (UID: \"1ee8f68b-fe59-4410-b1f6-861f71bffaf7\") "
Oct 31 00:18:07.667298 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:07.667177    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-maas-api-xks-ca\") pod \"1ee8f68b-fe59-4410-b1f6-861f71bffaf7\" (UID: \"1ee8f68b-fe59-4410-b1f6-861f71bffaf7\") "
Oct 31 00:18:07.667495 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:07.667463    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-maas-api-openshift-ca" pod "1ee8f68b-fe59-4410-b1f6-861f71bffaf7" (UID: "1ee8f68b-fe59-4410-b1f6-861f71bffaf7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:07.669384 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:07.669362    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-maas-api-xks-ca" pod "1ee8f68b-fe59-4410-b1f6-861f71bffaf7" (UID: "1ee8f68b-fe59-4410-b1f6-861f71bffaf7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:07.669474 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:07.669431    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-kube-api-access-tvjtl" pod "1ee8f68b-fe59-4410-b1f6-861f71bffaf7" (UID: "1ee8f68b-fe59-4410-b1f6-861f71bffaf7"). InnerVolumeSpecName "kube-api-access-tvjtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:07.768723 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:07.768688    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:18:07.768723 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:07.768718    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvjtl\" (UniqueName: \"kubernetes.io/projected/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-kube-api-access-tvjtl\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:18:07.768723 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:07.768729    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee8f68b-fe59-4410-b1f6-861f71bffaf7-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:18:08.399462 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:08.399427    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d"
Oct 31 00:18:08.399462 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:08.399446    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-pxw2d" event={"ID":"1ee8f68b-fe59-4410-b1f6-861f71bffaf7","Type":"ContainerDied","Data":"7c588e5b9e1d115e963df220d94d63ddb015e57bd8c83886321b67d3300d00d7"}
Oct 31 00:18:08.399713 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:08.399498    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c588e5b9e1d115e963df220d94d63ddb015e57bd8c83886321b67d3300d00d7"
Oct 31 00:18:17.097012 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:17.096976    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts"]
Oct 31 00:18:17.098167 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:17.098145    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-rszts"]
Oct 31 00:18:17.595379 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:18:17.595342    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:18:18.295933 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:18.295899    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62cbb295-c661-43ea-bc33-8cbae904fa7e" path="/var/lib/kubelet/pods/62cbb295-c661-43ea-bc33-8cbae904fa7e/volumes"
Oct 31 00:18:23.366623 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.366208    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88"]
Oct 31 00:18:23.367158 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.366709    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1ee8f68b-fe59-4410-b1f6-861f71bffaf7" containerName="revoke-keys"
Oct 31 00:18:23.367158 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.366728    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee8f68b-fe59-4410-b1f6-861f71bffaf7" containerName="revoke-keys"
Oct 31 00:18:23.367158 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.366771    5078 cpu_manager.go:414] "containerMap: removing container" podUID="fa64f8db-d461-41ce-be80-55c00e92c638" containerName="istio-proxy"
Oct 31 00:18:23.367158 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.366780    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa64f8db-d461-41ce-be80-55c00e92c638" containerName="istio-proxy"
Oct 31 00:18:23.367158 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.366885    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee8f68b-fe59-4410-b1f6-861f71bffaf7" containerName="revoke-keys"
Oct 31 00:18:23.370501 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.370478    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88"
Oct 31 00:18:23.423619 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.423414    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88"]
Oct 31 00:18:23.482621 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.473922    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:18:23.482621 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.474557    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:18:23.482621 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.474698    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnrc9\""
Oct 31 00:18:23.482621 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.474791    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:18:23.524621 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.521228    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-4mx88\" (UID: \"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88"
Oct 31 00:18:23.524621 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.521293    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6zjd\" (UniqueName: \"kubernetes.io/projected/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-kube-api-access-b6zjd\") pod \"maas-api-revoke-sub-0165d7d2-4mx88\" (UID: \"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88"
Oct 31 00:18:23.524621 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.521355    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-4mx88\" (UID: \"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88"
Oct 31 00:18:23.624679 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.624239    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-4mx88\" (UID: \"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88"
Oct 31 00:18:23.624679 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.624315    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6zjd\" (UniqueName: \"kubernetes.io/projected/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-kube-api-access-b6zjd\") pod \"maas-api-revoke-sub-0165d7d2-4mx88\" (UID: \"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88"
Oct 31 00:18:23.624679 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.624395    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-4mx88\" (UID: \"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88"
Oct 31 00:18:23.633617 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.630194    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-4mx88\" (UID: \"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88"
Oct 31 00:18:23.639274 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.639101    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-4mx88\" (UID: \"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88"
Oct 31 00:18:23.715558 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.715515    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6zjd\" (UniqueName: \"kubernetes.io/projected/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-kube-api-access-b6zjd\") pod \"maas-api-revoke-sub-0165d7d2-4mx88\" (UID: \"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88"
Oct 31 00:18:23.997673 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:23.996731    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88"
Oct 31 00:18:24.755166 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:24.755059    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88"]
Oct 31 00:18:25.669907 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:25.669869    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88" event={"ID":"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7","Type":"ContainerStarted","Data":"0b12ff67dc92f0fa483e65840e6c2dd2cb8d1dbe065f5517261de8484bb59151"}
Oct 31 00:18:25.670089 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:25.669925    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88" event={"ID":"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7","Type":"ContainerStarted","Data":"11da8dbce1d3347ecdfc4c7b981aef354b1fa016833ba64f96d8aa74dc8a9fac"}
Oct 31 00:18:25.784408 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:25.784357    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88" podStartSLOduration=2.784343883 podStartE2EDuration="2.784343883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:25.783211998 +0000 UTC m=+4291.671118362" watchObservedRunningTime="2026-10-31 00:18:25.784343883 +0000 UTC m=+4291.672250239"
Oct 31 00:18:26.675685 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:26.675650    5078 generic.go:356] "Generic (PLEG): container finished" podID="aaceeb43-f0b5-41fe-80f9-a5f8d45413e7" containerID="0b12ff67dc92f0fa483e65840e6c2dd2cb8d1dbe065f5517261de8484bb59151" exitCode=0
Oct 31 00:18:26.675847 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:26.675713    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88" event={"ID":"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7","Type":"ContainerDied","Data":"0b12ff67dc92f0fa483e65840e6c2dd2cb8d1dbe065f5517261de8484bb59151"}
Oct 31 00:18:27.803933 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:27.803906    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88"
Oct 31 00:18:27.834224 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:27.834193    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-maas-api-openshift-ca\") pod \"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7\" (UID: \"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7\") "
Oct 31 00:18:27.834408 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:27.834234    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-maas-api-xks-ca\") pod \"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7\" (UID: \"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7\") "
Oct 31 00:18:27.834408 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:27.834263    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-kube-api-access-b6zjd\" (UniqueName: \"kubernetes.io/projected/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-kube-api-access-b6zjd\") pod \"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7\" (UID: \"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7\") "
Oct 31 00:18:27.834629 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:27.834579    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-maas-api-openshift-ca" pod "aaceeb43-f0b5-41fe-80f9-a5f8d45413e7" (UID: "aaceeb43-f0b5-41fe-80f9-a5f8d45413e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:27.836433 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:27.836400    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-maas-api-xks-ca" pod "aaceeb43-f0b5-41fe-80f9-a5f8d45413e7" (UID: "aaceeb43-f0b5-41fe-80f9-a5f8d45413e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:27.836529 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:27.836439    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-kube-api-access-b6zjd" pod "aaceeb43-f0b5-41fe-80f9-a5f8d45413e7" (UID: "aaceeb43-f0b5-41fe-80f9-a5f8d45413e7"). InnerVolumeSpecName "kube-api-access-b6zjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:27.935036 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:27.934967    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6zjd\" (UniqueName: \"kubernetes.io/projected/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-kube-api-access-b6zjd\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:18:27.935036 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:27.935001    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:18:27.935036 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:27.935020    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aaceeb43-f0b5-41fe-80f9-a5f8d45413e7-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:18:28.686703 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:28.686621    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88"
Oct 31 00:18:28.686703 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:28.686651    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-4mx88" event={"ID":"aaceeb43-f0b5-41fe-80f9-a5f8d45413e7","Type":"ContainerDied","Data":"11da8dbce1d3347ecdfc4c7b981aef354b1fa016833ba64f96d8aa74dc8a9fac"}
Oct 31 00:18:28.686703 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:28.686701    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11da8dbce1d3347ecdfc4c7b981aef354b1fa016833ba64f96d8aa74dc8a9fac"
Oct 31 00:18:35.068424 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.068385    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf"]
Oct 31 00:18:35.068906 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.068750    5078 cpu_manager.go:414] "containerMap: removing container" podUID="aaceeb43-f0b5-41fe-80f9-a5f8d45413e7" containerName="revoke-keys"
Oct 31 00:18:35.068906 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.068766    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaceeb43-f0b5-41fe-80f9-a5f8d45413e7" containerName="revoke-keys"
Oct 31 00:18:35.068906 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.068827    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaceeb43-f0b5-41fe-80f9-a5f8d45413e7" containerName="revoke-keys"
Oct 31 00:18:35.071474 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.071458    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf"
Oct 31 00:18:35.100039 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.100006    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg6bq\" (UniqueName: \"kubernetes.io/projected/1c336490-f1f4-4868-922b-2f4486441174-kube-api-access-bg6bq\") pod \"maas-api-revoke-sub-6b19338f-fcbjf\" (UID: \"1c336490-f1f4-4868-922b-2f4486441174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf"
Oct 31 00:18:35.100204 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.100127    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c336490-f1f4-4868-922b-2f4486441174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-fcbjf\" (UID: \"1c336490-f1f4-4868-922b-2f4486441174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf"
Oct 31 00:18:35.100204 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.100193    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c336490-f1f4-4868-922b-2f4486441174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-fcbjf\" (UID: \"1c336490-f1f4-4868-922b-2f4486441174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf"
Oct 31 00:18:35.175781 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.175731    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 00:18:35.175781 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.175749    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:18:35.175781 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.175776    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:18:35.175781 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.175776    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hnrc9\""
Oct 31 00:18:35.177400 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.177377    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf"]
Oct 31 00:18:35.201699 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.201674    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c336490-f1f4-4868-922b-2f4486441174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-fcbjf\" (UID: \"1c336490-f1f4-4868-922b-2f4486441174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf"
Oct 31 00:18:35.201843 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.201708    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c336490-f1f4-4868-922b-2f4486441174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-fcbjf\" (UID: \"1c336490-f1f4-4868-922b-2f4486441174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf"
Oct 31 00:18:35.201843 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.201766    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg6bq\" (UniqueName: \"kubernetes.io/projected/1c336490-f1f4-4868-922b-2f4486441174-kube-api-access-bg6bq\") pod \"maas-api-revoke-sub-6b19338f-fcbjf\" (UID: \"1c336490-f1f4-4868-922b-2f4486441174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf"
Oct 31 00:18:35.202384 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.202355    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c336490-f1f4-4868-922b-2f4486441174-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-fcbjf\" (UID: \"1c336490-f1f4-4868-922b-2f4486441174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf"
Oct 31 00:18:35.204135 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.204115    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c336490-f1f4-4868-922b-2f4486441174-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-fcbjf\" (UID: \"1c336490-f1f4-4868-922b-2f4486441174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf"
Oct 31 00:18:35.259260 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.259235    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg6bq\" (UniqueName: \"kubernetes.io/projected/1c336490-f1f4-4868-922b-2f4486441174-kube-api-access-bg6bq\") pod \"maas-api-revoke-sub-6b19338f-fcbjf\" (UID: \"1c336490-f1f4-4868-922b-2f4486441174\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf"
Oct 31 00:18:35.383917 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.383842    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf"
Oct 31 00:18:35.756917 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:35.756889    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf"]
Oct 31 00:18:35.759507 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:18:35.759479    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c336490_f1f4_4868_922b_2f4486441174.slice/crio-4efa4c9104f17e9168cf32fa63c4fff4d6efe3b239e136fd3b055e6d5ccf929a WatchSource:0}: Error finding container 4efa4c9104f17e9168cf32fa63c4fff4d6efe3b239e136fd3b055e6d5ccf929a: Status 404 returned error can't find the container with id 4efa4c9104f17e9168cf32fa63c4fff4d6efe3b239e136fd3b055e6d5ccf929a
Oct 31 00:18:36.722414 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:36.722383    5078 generic.go:356] "Generic (PLEG): container finished" podID="1c336490-f1f4-4868-922b-2f4486441174" containerID="9f8d451c90da2454bd53ccd7fdf6132257ecbff823f1026d3112df51651f5482" exitCode=0
Oct 31 00:18:36.722823 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:36.722430    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf" event={"ID":"1c336490-f1f4-4868-922b-2f4486441174","Type":"ContainerDied","Data":"9f8d451c90da2454bd53ccd7fdf6132257ecbff823f1026d3112df51651f5482"}
Oct 31 00:18:36.722823 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:36.722477    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf" event={"ID":"1c336490-f1f4-4868-922b-2f4486441174","Type":"ContainerStarted","Data":"4efa4c9104f17e9168cf32fa63c4fff4d6efe3b239e136fd3b055e6d5ccf929a"}
Oct 31 00:18:37.856870 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:37.856847    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf"
Oct 31 00:18:37.925443 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:37.925421    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c336490-f1f4-4868-922b-2f4486441174-kube-api-access-bg6bq\" (UniqueName: \"kubernetes.io/projected/1c336490-f1f4-4868-922b-2f4486441174-kube-api-access-bg6bq\") pod \"1c336490-f1f4-4868-922b-2f4486441174\" (UID: \"1c336490-f1f4-4868-922b-2f4486441174\") "
Oct 31 00:18:37.925624 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:37.925475    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c336490-f1f4-4868-922b-2f4486441174-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c336490-f1f4-4868-922b-2f4486441174-maas-api-openshift-ca\") pod \"1c336490-f1f4-4868-922b-2f4486441174\" (UID: \"1c336490-f1f4-4868-922b-2f4486441174\") "
Oct 31 00:18:37.925624 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:37.925498    5078 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c336490-f1f4-4868-922b-2f4486441174-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c336490-f1f4-4868-922b-2f4486441174-maas-api-xks-ca\") pod \"1c336490-f1f4-4868-922b-2f4486441174\" (UID: \"1c336490-f1f4-4868-922b-2f4486441174\") "
Oct 31 00:18:37.925848 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:37.925825    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c336490-f1f4-4868-922b-2f4486441174-maas-api-openshift-ca" pod "1c336490-f1f4-4868-922b-2f4486441174" (UID: "1c336490-f1f4-4868-922b-2f4486441174"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:37.927752 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:37.927729    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c336490-f1f4-4868-922b-2f4486441174-maas-api-xks-ca" pod "1c336490-f1f4-4868-922b-2f4486441174" (UID: "1c336490-f1f4-4868-922b-2f4486441174"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:37.927833 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:37.927775    5078 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c336490-f1f4-4868-922b-2f4486441174-kube-api-access-bg6bq" pod "1c336490-f1f4-4868-922b-2f4486441174" (UID: "1c336490-f1f4-4868-922b-2f4486441174"). InnerVolumeSpecName "kube-api-access-bg6bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:38.026177 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:38.026101    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c336490-f1f4-4868-922b-2f4486441174-maas-api-openshift-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:18:38.026177 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:38.026127    5078 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c336490-f1f4-4868-922b-2f4486441174-maas-api-xks-ca\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:18:38.026177 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:38.026137    5078 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg6bq\" (UniqueName: \"kubernetes.io/projected/1c336490-f1f4-4868-922b-2f4486441174-kube-api-access-bg6bq\") on node \"ip-10-0-130-205.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:18:38.732881 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:38.732793    5078 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf"
Oct 31 00:18:38.732881 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:38.732843    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-fcbjf" event={"ID":"1c336490-f1f4-4868-922b-2f4486441174","Type":"ContainerDied","Data":"4efa4c9104f17e9168cf32fa63c4fff4d6efe3b239e136fd3b055e6d5ccf929a"}
Oct 31 00:18:38.733103 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:38.732901    5078 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4efa4c9104f17e9168cf32fa63c4fff4d6efe3b239e136fd3b055e6d5ccf929a"
Oct 31 00:18:41.052896 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:41.052864    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq"]
Oct 31 00:18:41.102842 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:41.102808    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-6pvbq"]
Oct 31 00:18:42.296228 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:42.296194    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af26640a-3be4-4f5a-bb6a-897b5cea489a" path="/var/lib/kubelet/pods/af26640a-3be4-4f5a-bb6a-897b5cea489a/volumes"
Oct 31 00:18:47.589171 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:18:47.589135    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:18:59.563907 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:18:59.563877    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bbf6fc889-972gn_a43eb438-d90f-4886-bba3-c2b4531028cf/manager/0.log"
Oct 31 00:19:01.762175 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:01.762145    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-4mx88_aaceeb43-f0b5-41fe-80f9-a5f8d45413e7/revoke-keys/0.log"
Oct 31 00:19:01.983038 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:01.983003    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-60ed7e88-jtxf4_4f13cdf0-9e63-4588-99cb-76f47ed0555c/revoke-keys/0.log"
Oct 31 00:19:02.200935 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:02.200909    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-fcbjf_1c336490-f1f4-4868-922b-2f4486441174/revoke-keys/0.log"
Oct 31 00:19:02.414967 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:02.414930    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-pxw2d_1ee8f68b-fe59-4410-b1f6-861f71bffaf7/revoke-keys/0.log"
Oct 31 00:19:06.833174 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:06.833076    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rvtqr_4b80c6f1-0c87-4ae4-9ffc-c24b97741f93/discovery/0.log"
Oct 31 00:19:07.202802 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:07.202724    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-bd9cf9d49-6tcdv_a8c68f95-3c84-4af1-9d26-8eedf2fc8f79/kube-auth-proxy/0.log"
Oct 31 00:19:07.661067 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:07.661021    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-42wvb_c76a2fb3-5cc9-4e1d-bc93-c8acf8723819/istio-proxy/0.log"
Oct 31 00:19:07.885542 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:07.885508    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-4662g_9a60053e-3454-4ef1-9b40-9194f19b840d/istio-proxy/0.log"
Oct 31 00:19:08.091026 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:08.091000    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-4g4vj_9af70286-d197-473c-be7d-9333a83d1c4d/istio-proxy/0.log"
Oct 31 00:19:08.320635 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:08.320586    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-659rv_c9e4b867-ea81-4fc3-91c4-ce1129b13d3f/istio-proxy/0.log"
Oct 31 00:19:11.051802 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:11.051768    5078 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4"]
Oct 31 00:19:11.101520 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:11.101494    5078 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60ed7e88-jtxf4"]
Oct 31 00:19:12.209897 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:12.209866    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-j6k6v_82a135d9-df91-4118-a6fe-feb2d0f4aeba/main/0.log"
Oct 31 00:19:12.296008 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:12.295978    5078 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f13cdf0-9e63-4588-99cb-76f47ed0555c" path="/var/lib/kubelet/pods/4f13cdf0-9e63-4588-99cb-76f47ed0555c/volumes"
Oct 31 00:19:12.876268 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:12.876228    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qwb8z_f70cda85-5da1-43d7-b94b-175bbaec28a5/main/0.log"
Oct 31 00:19:13.316339 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:13.316308    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn_7160e014-9b36-4fc3-afc6-82aa41241b3d/main/0.log"
Oct 31 00:19:17.589162 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:19:17.589121    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:19:24.613892 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:24.613864    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bbf6fc889-972gn_a43eb438-d90f-4886-bba3-c2b4531028cf/manager/0.log"
Oct 31 00:19:26.617243 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:26.617212    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-4mx88_aaceeb43-f0b5-41fe-80f9-a5f8d45413e7/revoke-keys/0.log"
Oct 31 00:19:26.834896 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:26.834868    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-fcbjf_1c336490-f1f4-4868-922b-2f4486441174/revoke-keys/0.log"
Oct 31 00:19:27.064360 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:27.064322    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-pxw2d_1ee8f68b-fe59-4410-b1f6-861f71bffaf7/revoke-keys/0.log"
Oct 31 00:19:30.801245 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:30.801213    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rvtqr_4b80c6f1-0c87-4ae4-9ffc-c24b97741f93/discovery/0.log"
Oct 31 00:19:31.116007 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:31.115914    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-bd9cf9d49-6tcdv_a8c68f95-3c84-4af1-9d26-8eedf2fc8f79/kube-auth-proxy/0.log"
Oct 31 00:19:31.551885 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:31.551860    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-42wvb_c76a2fb3-5cc9-4e1d-bc93-c8acf8723819/istio-proxy/0.log"
Oct 31 00:19:31.780055 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:31.780024    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-4662g_9a60053e-3454-4ef1-9b40-9194f19b840d/istio-proxy/0.log"
Oct 31 00:19:32.002322 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:32.002248    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-4g4vj_9af70286-d197-473c-be7d-9333a83d1c4d/istio-proxy/0.log"
Oct 31 00:19:32.215942 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:32.215915    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-659rv_c9e4b867-ea81-4fc3-91c4-ce1129b13d3f/istio-proxy/0.log"
Oct 31 00:19:35.586873 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:35.586844    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-j6k6v_82a135d9-df91-4118-a6fe-feb2d0f4aeba/main/0.log"
Oct 31 00:19:36.291026 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:36.290990    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qwb8z_f70cda85-5da1-43d7-b94b-175bbaec28a5/main/0.log"
Oct 31 00:19:36.728480 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:19:36.728393    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-dzlrn_7160e014-9b36-4fc3-afc6-82aa41241b3d/main/0.log"
Oct 31 00:19:47.590679 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:19:47.590642    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:20:04.665973 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:04.665941    5078 scope.go:122] "RemoveContainer" containerID="133037770bcf6c89780bd3c0c36d6dc290b37d6db9e5ba7f17a6369a1aa56cc4"
Oct 31 00:20:04.674519 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:04.674496    5078 scope.go:122] "RemoveContainer" containerID="320d4a5021751125ecddb1e96fe70862d233c06b4fd55f8e690841f010513b4e"
Oct 31 00:20:07.406148 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:07.406113    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bbf6fc889-972gn_a43eb438-d90f-4886-bba3-c2b4531028cf/manager/0.log"
Oct 31 00:20:07.908943 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:07.908910    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bbf6fc889-972gn_a43eb438-d90f-4886-bba3-c2b4531028cf/manager/0.log"
Oct 31 00:20:17.590438 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:20:17.590398    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:20:47.433000 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:47.432966    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bbf6fc889-972gn_a43eb438-d90f-4886-bba3-c2b4531028cf/manager/0.log"
Oct 31 00:20:47.590278 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:20:47.590240    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:20:48.010712 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:48.010674    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7bbf6fc889-972gn_a43eb438-d90f-4886-bba3-c2b4531028cf/manager/0.log"
Oct 31 00:20:49.341312 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:49.341274    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d5xbs/must-gather-wl8x2"]
Oct 31 00:20:49.341739 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:49.341582    5078 cpu_manager.go:414] "containerMap: removing container" podUID="1c336490-f1f4-4868-922b-2f4486441174" containerName="revoke-keys"
Oct 31 00:20:49.341739 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:49.341607    5078 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c336490-f1f4-4868-922b-2f4486441174" containerName="revoke-keys"
Oct 31 00:20:49.341739 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:49.341684    5078 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c336490-f1f4-4868-922b-2f4486441174" containerName="revoke-keys"
Oct 31 00:20:49.344835 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:49.344817    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d5xbs/must-gather-wl8x2"
Oct 31 00:20:49.396952 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:49.396908    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d5xbs\"/\"openshift-service-ca.crt\""
Oct 31 00:20:49.397135 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:49.396975    5078 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-d5xbs\"/\"kube-root-ca.crt\""
Oct 31 00:20:49.420356 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:49.420319    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzr2h\" (UniqueName: \"kubernetes.io/projected/69241480-7d5b-4f4f-bec3-3cf47c4d9806-kube-api-access-gzr2h\") pod \"must-gather-wl8x2\" (UID: \"69241480-7d5b-4f4f-bec3-3cf47c4d9806\") " pod="openshift-must-gather-d5xbs/must-gather-wl8x2"
Oct 31 00:20:49.420536 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:49.420485    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/69241480-7d5b-4f4f-bec3-3cf47c4d9806-must-gather-output\") pod \"must-gather-wl8x2\" (UID: \"69241480-7d5b-4f4f-bec3-3cf47c4d9806\") " pod="openshift-must-gather-d5xbs/must-gather-wl8x2"
Oct 31 00:20:49.446890 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:49.446850    5078 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-d5xbs\"/\"default-dockercfg-xrnc4\""
Oct 31 00:20:49.448423 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:49.448400    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d5xbs/must-gather-wl8x2"]
Oct 31 00:20:49.521649 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:49.521589    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzr2h\" (UniqueName: \"kubernetes.io/projected/69241480-7d5b-4f4f-bec3-3cf47c4d9806-kube-api-access-gzr2h\") pod \"must-gather-wl8x2\" (UID: \"69241480-7d5b-4f4f-bec3-3cf47c4d9806\") " pod="openshift-must-gather-d5xbs/must-gather-wl8x2"
Oct 31 00:20:49.521838 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:49.521691    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/69241480-7d5b-4f4f-bec3-3cf47c4d9806-must-gather-output\") pod \"must-gather-wl8x2\" (UID: \"69241480-7d5b-4f4f-bec3-3cf47c4d9806\") " pod="openshift-must-gather-d5xbs/must-gather-wl8x2"
Oct 31 00:20:49.522021 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:49.522000    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/69241480-7d5b-4f4f-bec3-3cf47c4d9806-must-gather-output\") pod \"must-gather-wl8x2\" (UID: \"69241480-7d5b-4f4f-bec3-3cf47c4d9806\") " pod="openshift-must-gather-d5xbs/must-gather-wl8x2"
Oct 31 00:20:49.579830 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:49.579797    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzr2h\" (UniqueName: \"kubernetes.io/projected/69241480-7d5b-4f4f-bec3-3cf47c4d9806-kube-api-access-gzr2h\") pod \"must-gather-wl8x2\" (UID: \"69241480-7d5b-4f4f-bec3-3cf47c4d9806\") " pod="openshift-must-gather-d5xbs/must-gather-wl8x2"
Oct 31 00:20:49.658386 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:49.658304    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d5xbs/must-gather-wl8x2"
Oct 31 00:20:50.027246 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:50.027093    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d5xbs/must-gather-wl8x2"]
Oct 31 00:20:50.029453 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:20:50.029426    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69241480_7d5b_4f4f_bec3_3cf47c4d9806.slice/crio-c355fe44d31c791b642893f8dc807754f83f813d41d553f9c45ddee714d91303 WatchSource:0}: Error finding container c355fe44d31c791b642893f8dc807754f83f813d41d553f9c45ddee714d91303: Status 404 returned error can't find the container with id c355fe44d31c791b642893f8dc807754f83f813d41d553f9c45ddee714d91303
Oct 31 00:20:50.031253 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:50.031236    5078 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:20:50.432257 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:50.432223    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d5xbs/must-gather-wl8x2" event={"ID":"69241480-7d5b-4f4f-bec3-3cf47c4d9806","Type":"ContainerStarted","Data":"c355fe44d31c791b642893f8dc807754f83f813d41d553f9c45ddee714d91303"}
Oct 31 00:20:52.448583 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:52.448537    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d5xbs/must-gather-wl8x2" event={"ID":"69241480-7d5b-4f4f-bec3-3cf47c4d9806","Type":"ContainerStarted","Data":"c036649d3d40388aef0b009ddae19a736b6c1a65f289059bfdd139fb359c6486"}
Oct 31 00:20:52.449110 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:52.448590    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d5xbs/must-gather-wl8x2" event={"ID":"69241480-7d5b-4f4f-bec3-3cf47c4d9806","Type":"ContainerStarted","Data":"72be674362c1bf092330e9958caa1e828115d4a5c083ff342745831e1c4852d2"}
Oct 31 00:20:52.563486 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:52.563428    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-d5xbs/must-gather-wl8x2" podStartSLOduration=2.208320153 podStartE2EDuration="3.563413382s" totalImagesPullingTime="1.355093229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:52.562109414 +0000 UTC m=+4438.450015766" watchObservedRunningTime="2026-10-31 00:20:52.563413382 +0000 UTC m=+4438.451319811"
Oct 31 00:20:59.624276 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:20:59.624239    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cthfm_2e942185-68e3-4d57-9f42-49d2965b41e4/global-pull-secret-syncer/0.log"
Oct 31 00:21:01.991767 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:01.991732    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-fg8fg_7e14c04d-b583-4f0a-840c-d493b2a2bbaf/konnectivity-agent/0.log"
Oct 31 00:21:03.273700 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:03.273671    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-205.us-west-1.compute.internal_a1b66be07f6b0e09e908458ab3790deb/haproxy/0.log"
Oct 31 00:21:04.703003 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:04.702966    5078 scope.go:122] "RemoveContainer" containerID="7512928f74b7c0d979fdae94ad91c6a15281e636e5a74ca2bae0d3e7bce399c2"
Oct 31 00:21:04.728196 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:04.728161    5078 scope.go:122] "RemoveContainer" containerID="8b2ac9cfdf2a7410d3db13ff09e2c17edff192eebaa275a6de0bb3452df4f60c"
Oct 31 00:21:08.310141 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.310103    5078 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"]
Oct 31 00:21:08.317348 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.317320    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:08.406046 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.406011    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"]
Oct 31 00:21:08.408292 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.408258    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aaee0362-4491-48fd-aa83-34e92c9705f9-podres\") pod \"perf-node-gather-daemonset-xdp5f\" (UID: \"aaee0362-4491-48fd-aa83-34e92c9705f9\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:08.408456 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.408391    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aaee0362-4491-48fd-aa83-34e92c9705f9-lib-modules\") pod \"perf-node-gather-daemonset-xdp5f\" (UID: \"aaee0362-4491-48fd-aa83-34e92c9705f9\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:08.408456 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.408448    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aaee0362-4491-48fd-aa83-34e92c9705f9-sys\") pod \"perf-node-gather-daemonset-xdp5f\" (UID: \"aaee0362-4491-48fd-aa83-34e92c9705f9\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:08.408585 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.408539    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzmff\" (UniqueName: \"kubernetes.io/projected/aaee0362-4491-48fd-aa83-34e92c9705f9-kube-api-access-fzmff\") pod \"perf-node-gather-daemonset-xdp5f\" (UID: \"aaee0362-4491-48fd-aa83-34e92c9705f9\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:08.408690 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.408627    5078 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aaee0362-4491-48fd-aa83-34e92c9705f9-proc\") pod \"perf-node-gather-daemonset-xdp5f\" (UID: \"aaee0362-4491-48fd-aa83-34e92c9705f9\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:08.509780 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.509745    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aaee0362-4491-48fd-aa83-34e92c9705f9-lib-modules\") pod \"perf-node-gather-daemonset-xdp5f\" (UID: \"aaee0362-4491-48fd-aa83-34e92c9705f9\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:08.509968 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.509788    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aaee0362-4491-48fd-aa83-34e92c9705f9-sys\") pod \"perf-node-gather-daemonset-xdp5f\" (UID: \"aaee0362-4491-48fd-aa83-34e92c9705f9\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:08.509968 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.509822    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzmff\" (UniqueName: \"kubernetes.io/projected/aaee0362-4491-48fd-aa83-34e92c9705f9-kube-api-access-fzmff\") pod \"perf-node-gather-daemonset-xdp5f\" (UID: \"aaee0362-4491-48fd-aa83-34e92c9705f9\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:08.509968 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.509858    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aaee0362-4491-48fd-aa83-34e92c9705f9-proc\") pod \"perf-node-gather-daemonset-xdp5f\" (UID: \"aaee0362-4491-48fd-aa83-34e92c9705f9\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:08.509968 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.509877    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aaee0362-4491-48fd-aa83-34e92c9705f9-lib-modules\") pod \"perf-node-gather-daemonset-xdp5f\" (UID: \"aaee0362-4491-48fd-aa83-34e92c9705f9\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:08.509968 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.509885    5078 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aaee0362-4491-48fd-aa83-34e92c9705f9-podres\") pod \"perf-node-gather-daemonset-xdp5f\" (UID: \"aaee0362-4491-48fd-aa83-34e92c9705f9\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:08.509968 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.509931    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aaee0362-4491-48fd-aa83-34e92c9705f9-sys\") pod \"perf-node-gather-daemonset-xdp5f\" (UID: \"aaee0362-4491-48fd-aa83-34e92c9705f9\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:08.509968 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.509935    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aaee0362-4491-48fd-aa83-34e92c9705f9-podres\") pod \"perf-node-gather-daemonset-xdp5f\" (UID: \"aaee0362-4491-48fd-aa83-34e92c9705f9\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:08.509968 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.509962    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aaee0362-4491-48fd-aa83-34e92c9705f9-proc\") pod \"perf-node-gather-daemonset-xdp5f\" (UID: \"aaee0362-4491-48fd-aa83-34e92c9705f9\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:08.566542 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.566474    5078 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzmff\" (UniqueName: \"kubernetes.io/projected/aaee0362-4491-48fd-aa83-34e92c9705f9-kube-api-access-fzmff\") pod \"perf-node-gather-daemonset-xdp5f\" (UID: \"aaee0362-4491-48fd-aa83-34e92c9705f9\") " pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:08.632760 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:08.632727    5078 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:09.006239 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:09.006068    5078 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"]
Oct 31 00:21:09.008968 ip-10-0-130-205 kubenswrapper[5078]: W1031 00:21:09.008944    5078 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podaaee0362_4491_48fd_aa83_34e92c9705f9.slice/crio-dd2afc3cd8781bc12168597bfafb626f56b2dedff7e8a7849c620a2d7a16af09 WatchSource:0}: Error finding container dd2afc3cd8781bc12168597bfafb626f56b2dedff7e8a7849c620a2d7a16af09: Status 404 returned error can't find the container with id dd2afc3cd8781bc12168597bfafb626f56b2dedff7e8a7849c620a2d7a16af09
Oct 31 00:21:09.568231 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:09.568197    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f" event={"ID":"aaee0362-4491-48fd-aa83-34e92c9705f9","Type":"ContainerStarted","Data":"cc9f0d7e3828ec2af8dccc127171134270a66fb8c5a31b9106f55cba80186346"}
Oct 31 00:21:09.568703 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:09.568251    5078 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f" event={"ID":"aaee0362-4491-48fd-aa83-34e92c9705f9","Type":"ContainerStarted","Data":"dd2afc3cd8781bc12168597bfafb626f56b2dedff7e8a7849c620a2d7a16af09"}
Oct 31 00:21:09.568703 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:09.568346    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:09.682909 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:09.682858    5078 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f" podStartSLOduration=1.68284442 podStartE2EDuration="1.68284442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:09.680524152 +0000 UTC m=+4455.568430507" watchObservedRunningTime="2026-10-31 00:21:09.68284442 +0000 UTC m=+4455.570750773"
Oct 31 00:21:15.583312 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:15.583279    5078 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-d5xbs/perf-node-gather-daemonset-xdp5f"
Oct 31 00:21:17.595646 ip-10-0-130-205 kubenswrapper[5078]: E1031 00:21:17.595590    5078 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-tcf86" containerName="extractor"
Oct 31 00:21:19.877015 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:19.876988    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6bv5g_589b7d5b-70f8-440e-a171-3ba10ed5bf7a/node-exporter/0.log"
Oct 31 00:21:20.295256 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:20.295226    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6bv5g_589b7d5b-70f8-440e-a171-3ba10ed5bf7a/kube-rbac-proxy/0.log"
Oct 31 00:21:20.713676 ip-10-0-130-205 kubenswrapper[5078]: I1031 00:21:20.713586    5078 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6bv5g_589b7d5b-70f8-440e-a171-3ba10ed5bf7a/init-textfile/0.log"